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OVERSIGHT  PROCESS  OF  THE  MAJOR  AUTOMATED 
INFORMATION  SYSTEMS  REVIEW  COUNCIL 

EXECUTIVE  SUMMARY 


Introduction.  The  Major  Automated  Information  Systems  Review  Council  (MAISRC) 
is  chaired  by  the  Assistant  Secretary  of  Defense  (Command,  Control,  Communications, 
and  Intelligence)  and  is  composed  of  senior  representatives  from  the  Office  of  the 
Secretary  of  Defense.  The  MAISRC  has  traditionally  provided  oversight  for  DoD 
business-like,  major  (high-cost  or  of  special-interest)  automated  information  systems. 
However,  that  oversight  is  changing  to  include  other  major  automated  information 
systems,  such  as  command  and  control,  communications,  and  intelligence  systems,  that 
support  the  front-line  combatant.  During  FYs  1995  and  1996,  DoD  will  spend  about 
$3  billion  to  develop,  modernize,  and  operate  major  automated  information  systems. 

Objectives.  The  audit  objective  was  to  determine  whether  the  MAISRC  oversight 
process  was  effective  in  supporting  DoD's  information  management  goals  and 
initiatives.  We  also  evaluated  management  controls  related  to  the  MAISRC  oversight 
process. 

Audit  Results.  In  implementing  its  information  management  initiatives,  the  DoD  has 
changed  acquisition  methods  used  to  develop  or  modernize  major  automated 
information  systems.  However,  the  MAISRC  oversight  process  for  DoD’s  major 
automated  information  systems  has  not  yet  been  similarly  reengineered.  Revising  the 
MAISRC  process  would  provide  DoD  the  needed  assurance  that  the  oversight 
effectively  supports  DoD  information  management  goals.  Details  are  in  Part  H .  The 
audit  identified  material  weaknesses  in  the  MAISRC  management  control  process 
involving  the  validation  of  costs  and  benefits;  when  and  how  systems  developed  in 
increments  would  be  tested;  and  oversight  of  command  and  control,  communications, 
and  intelligence  automated  information  systems.  The  management  controls  assessed  are 
described  in  Part  I,  and  the  weaknesses  are  discussed  in  Part  II. 

Implementation  of  the  recommendations  will  result  in  a  more  effective  MAISRC 
process  that  supports  DoD  information  management  goals.  Management  will  be  able  to 
determine  monetary  benefits  when  the  recommendation  on  validated  costs  and  benefits 
is  implemented.  Appendix  E  summarizes  other  potential  benefits  resulting  from  the 
audit. 

Summary  of  Recommendations.  We  recommend  revising  DoD  regulations  to  specify 
procedures  that  will  involve  the  MAISRC  in  ongoing  DoD  Corporate  Information 
Management  Initiative  efforts  and  to  specify  procedures  for  the  MAISRC  to  use  in 
performing  independent  assessments  of  the  selection  and  development  of  major 
automated  information  systems  that  will  be  used  on  a  DoD-wide  basis.  We  also 
recommend  revising  guidance  to  specify  procedures  for  performing  operational  testing 
and  validating  costs  and  benefits  of  automated  information  systems  developed  in 
increments.  We  further  recommend  that  key,  minimum  documentation  be  determined 
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for  each  type  of  MAISRC  review  and  that  oversight  responsibility  be  clarified  for 
command  and  control,  communications,  and  intelligence  major  automated  information 
systems. 

Management  Comments.  The  Deputy  Assistant  Secretary  (Acquisition),  responding 
for  the  Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence),  concurred  with  most  of  the  recommendations.  Management  stated  that 
guidance  would  be  revised  to  specify  procedures  for  performing  operational  testing  and 
for  validating  costs  and  benefits  of  automated  information  systems  developed  in 
increments,  to  establish  key  documentation  for  MAISRC  reviews,  and  to  clarify 
oversight  responsibilities  for  command  and  control  major  automated  information 
systems.  Also,  management  would  consider  clarifying  procedures  for  the  oversight  of 
incrementally  developed  systems.  Management  did  not  agree  that  procedures  should  be 
specified  to  actively  involve  the  MAISRC  staff  in  Corporate  Information  Management 
Initiative  efforts,  including  the  selection  and  development  of  migratory  systems.* 

Audit  Response.  As  a  result  of  management  comments,  we  revised  the 
recommendations  concerning  the  oversight  of  incrementally  developed  systems  and 
enhanced  cost  and  benefit  validations  to  clarify  our  intent.  We  believe  management 
needs  to  implement  further  action  to  improve  the  MAISRC  process  including  more 
interaction  between  staff  elements,  possibly  along  the  lines  of  the  Integrated  Product 
team  approach  now  being  introduced  in  the  weapons  system  area.  A  discussion  of 
management  comments  and  audit  responses  is  in  Part  II  of  the  report.  The  complete 
text  of  management  comments  is  in  Part  IV.  We  ask  that  the  Assistant  Secretary  of 
Defense  (Command,  Control,  Communications,  and  Intelligence)  provide  additional 
comments  on  the  final  report  by  August  30,  1995. 


*An  existing  or  planned  and  approved  AIS  that  has  been  designated  to  support  a 
functional  .process  on  a  DoD-wide  basis. 
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Introduction 


Background 


Major  Automated  Information  Systems.  Since  1978,  the  Major  Automated 
Information  Systems  Review  Council  (MAISRC)  has  represented  the  Secretary 
of  Defense  in  performing  reviews  of  high-cost  or  special-interest  (major) 
automated  information  systems  (AISs).  An  AIS  is  any  combination  of 
information,  computer,  telecommunications,  other  information  technology,  and 
personnel  resources  that  collect,  record,  process,  store,  communicate,  retrieve, 
and  display  information.  The  primary  purpose  of  a  MAISRC  review  is  to 
determine  whether  the  acquisition  or  development  of  a  major  AIS  should  be 
continued,  redirected,  or  terminated.  Appendix  A  specifies  the  criteria  for  a 
major  AIS. 

MAISRC  Members.  The  MAISRC  is  chaired  by  the  Assistant  Secretary  of 
Defense  (Command,  Control,  Communications,  and  Intelligence)  who  also 
serves  as  the  milestone  decision  authority.  Further,  the  MAISRC  is  composed 
of  designated  Office  of  the  Secretary  of  Defense  Principal  Staff  Assistants, 
including  those  who  manage  the  functional  areas  supported  by  the  AISs  under 
review;  the  Joint  Staff  representative(s);  the  Senior  Acquisition  Authority  for 
the  AIS  program  under  review;  and  other  members  (see  Appendix  B)  selected 
by  the  Assistant  Secretary  of  Defense  (Command,  Control,  Communications, 
and  Intelligence). 

DoD's  Corporate  Information  Management  Initiative.  Driven  by  reduced 
Defense  budgets,  the  need  to  reduce  operating  and  support  costs,  and  the 
recognized  need  for  integrated  and  interoperable  AISs,  DoD  has  been  changing 
the  way  it  views,  develops,  and  uses  AISs.  DoD  is  changing  its  focus  from 
individual  AISs  developed  to  meet  Component-specific  requirements  to  AISs 
developed  to  meet  DoD-wide  requirements.  To  achieve  its  information 
management  goals,  in  1990  DoD  initiated  several  related  efforts  that  are 
collectively  called  the  Corporate  Information  Management  Initiative.  The 
ultimate  achievement  of  DoD  information  management  goals  is  based  on  four 
main  elements  of  the  Corporate  Information  Management  Initiative. 

o  Functional  process  improvement  focuses  on  using  information 
technology  to  redefine  functional  processes  to  be  more  directly  aligned  with 
desired  results. 

o  Migration  systems  are  those  AISs  designated  for  interim  use  for 
functional  processes  on  a  DoD-wide  basis  during  the  transition  from  numerous 
existing  (legacy)  and  nonstandard  AISs. 

o  Data  administration  involves  the  development  and  DoD-wide  use  of 
standard  data  definitions  and  the  ultimate  establishment  of  DoD  "corporate"  data 
bases. 


o  The  Defense  Information  Infrastructure  encompasses  information 
transfer  and  processing  resources  and  is  envisioned  to  connect  DoD  AISs  for 
mission  support,  command  and  control,  and  intelligence  through  voice,  data, 
imagery,  video,  and  multimedia  services. 
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Although  the  Corporate  Information  Management  Initiative  will  involve 
substantial  investment,  DoD  expects  its  implementation  to  result  in  long-term 
economies  and  efficiencies. 


Objectives 


The  audit  objective  was  to  determine  whether  the  MAISRC  oversight  process 
was  effective  in  supporting  DoD's  information  management  goals  and 
initiatives.  We  also  evaluated  management  controls  related  to  the  MAISRC 
oversight  process. 


Scope  and  Methodology 


We  evaluated  the  MAISRC  oversight  process  for  major  AISs.  We  reviewed 
DoD  guidelines,  requirements,  and  related  documentation  (dated  from 
March  1988  through  January  1995)  that  applied  to  AIS  life-cycle  management 
and  to  the  oversight  of  major  AISs.  We  discussed  MAISRC  review  procedures 
and  requirements  with  MAISRC  members  and  staff.  We  gathered  data  on  the 
53  major  AISs  (see  Appendix  C)  subject  to  MAISRC  oversight  as  of 
August  1994,  including  FYs  1994  and  1995  program  budgets,  estimated 
life-cycle  costs,  and  type  and  date  of  the  most  recent  oversight  review. 
Although  oversight  responsibility  for  15  of  the  53  major  AISs  was  delegated  to 
DoD  Components,  our  audit  focused  on  the  MAISRC  oversight  process  and  not 
the  oversight  process  used  by  the  Components.  Also,  we  coordinated  with  the 
responsible  Principal  Staff  Assistants  and  program  managers  of  major  AISs  to 
gain  their  perspectives  of  the  MAISRC  process. 

This  economy  and  efficiency  audit  was  performed  from  April  1994  through 
January  1995  in  accordance  with  auditing  standards  issued  by  the  Comptroller 
General  of  the  United  States  as  implemented  by  the  Inspector  General,  DoD. 
The  audit  included  such  tests  of  management  controls  as  were  considered 
necessary.  We  did  not  rely  on  computer-processed  data  or  statistical  sampling 
procedures  to  achieve  the  audit  objectives.  Appendix  F  lists  organizations 
visited  or  contacted  during  the  audit. 


Management  Controls 


DoD  Directive  5010.38,  "Internal  Management  Control  Program," 
April  14,  1987,  requires  every  DoD  organization  to  have  management  controls 
in  place  for  its  operations  and  to  perform  periodic  evaluations  of  those 
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controls.  The  MAISRC  is,  in  essence,  a  management  control.  We  reviewed 
the  MAISRC  process  and  the  self-evaluation  aspects  of  DoD  Directive  5010.38 
as  they  relate  to  that  process. 

Management  Controls  Reviewed.  We  reviewed  management  control 
procedures  used  to  identify  major  AISs,  documentation  of  MAISRC  reviews, 
methods  used  to  track  returns  on  investments  for  major  AISs,  and  MAISRC 
compliance  with  laws  and  regulations. 

Adequacy  of  Management  Controls.  We  identified  material  management 
control  weaknesses  in  that  the  MAISRC  did  not  consistently  validate  major  AIS 
costs  and  benefits;  establish  procedures  for  the  operational  testing  and 
evaluation  of  major  AISs  developed  in  increments;  or  determine  oversight 
responsibilities  for  command  and  control,  communications,  and  intelligence 
major  AISs.  See  the  finding  for  details.  Recommendations  4.,  5.,  and  8.,  if 
implemented,  will  correct  those  weaknesses.  Potential  monetary  benefits 
associated  with  validating  costs  and  benefits  can  be  determined  once  validations 
are  performed.  See  Appendix  E  for  all  benefits  associated  with  the  audit.  A 
copy  of  the  report  will  be  provided  to  the  senior  official  responsible  for 
management  controls  within  the  Office  of  the  Secretary  of  Defense. 

Adequacy  of  the  MAISRC  Self-Evaluation  of  Applicable  Management 
Controls.  Office  of  the  Assistant  Secretary  of  Defense  (Command,  Control, 
Communications,  and  Intelligence)  officials  did  not  identify  the  MAISRC  as  an 
assessable  unit.  Those  officials  did  not  identify  the  MAISRC  as  an  assessable 
unit  because  they  viewed  the  MAISRC  process  as  a  management  control  and 
believed  that  there  was  no  requirement  to  designate  portions  of  the  management 
control  structure  as  assessable  units.  We  agree  that  the  MAISRC  process 
constitutes  a  management  control,  but  that  is  all  the  more  reason  why  it  should 
be  considered  an  assessable  unit  and  periodically  evaluated  by  management. 


Prior  Audits  and  Other  Reviews 


Since  January  1990,  the  General  Accounting  Office;  the  Inspector  General, 
DoD;  and  the  Military  Department  audit  organizations  have  issued  186  reports 
concerning  management  of  AISs.  Appendix  D  lists  44  of  those  reports  that 
discuss  the  development  and  acquisition  of  AISs  subject  to  MAISRC  oversight. 
Summarized  below  are  reports  issued  by  the  Office  of  the  Secretary  of  Defense 
Process  Action  Team  for  Acquisition  Reform  and  by  the  Subcommittee  on 
Oversight  of  Government  Management,  Senate  Committee  on  Governmental 
Affairs.  Both  of  those  reports  discuss  the  need  for  more  effective  oversight  of 
major  AISs.  Also  summarized  below  is  a  General  Accounting  Office  report  that 
specifically  discusses  DoD  oversight  of  major  AISs. 
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"Reengineering  the  Acquisition  Oversight  and  Review  Process," 
December  9,  1994,  Office  of  the  Secretary  of  Defense  Process  Action  Team 
for  Acquisition  Reform.  The  report  concludes  that  DoD  needs  to  reengineer 
its  acquisition  oversight  and  review  process  to  make  it  more  effective  and 
efficient.  The  report  focuses  on  improving  the  oversight  and  review  of  weapon 
system  acquisitions  and  recommends  that  the  Assistant  Secretary  of  Defense 
(Command,  Control,  Communications,  and  Intelligence)  adapt  the  report’s 
recommendations,  as  appropriate,  to  the  MAISRC  review  and  oversight 
process.  We  reached  a  similar  conclusion,  as  discussed  in  Part  II  of  this  report, 
in  that  the  amount  of  documentation  required  for  milestone  reviews  was 
unnecessarily  burdensome.  As  a  result  of  the  report,  the  Under  Secretary  of 
Defense  (Acquisition  and  Technology)  determined  that  the  AIS  acquisition 
process  should  be  integrated  with  the  weapon  system  acquisition  process  to  the 
maximum  practicable  extent.  A  separate  report  on  a  plan  to  accomplish  that 
integration  is  to  be  provided  to  the  Under  Secretary  and  the  Assistant  Secretary 
of  Defense  (Command,  Control,  Communications,  and  Intelligence)  by 
July  1995. 

"Computer  Chaos:  Billions  Wasted  Buying  Federal  Computer  Systems," 
October  12,  1994,  Subcommittee  on  Oversight  of  Government 

Management,  Senate  Committee  on  Governmental  Affairs.  The  report 
identifies  Federal  agency  weaknesses  in  procuring  computer  systems.  The 
report  recommendations  focus  on  changing  the  process  for  buying  major  AISs 
and  emphasize  the  need  for  earlier  and  more  effective  oversight  of  those  AISs. 
Although  the  recommendations  were  not  specifically  directed  to  DoD,  the 
Secretary  of  Defense  was  requested  to  comment  by  December  17,  1994.  The 
Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence)  concurred  with  all  but  one  recommendation.  The  Assistant 
Secretary  requested  reconsideration  of  the  recommendation  to  reevaluate  and 
halt  existing  procurements  until  the  acquisition  process  is  improved. 

"Automated  Information  Systems,  Defense's  Oversight  Process  Should  Be 
Improved,"  April  1990,  General  Accounting  Office  Report 
No.  IMTEC-90-36  (Office  of  the  Secretary  of  Defense  Case  No.  8328).  The 
report  concludes  that  the  Office  of  the  Secretary  of  Defense  had  not  been 
effective  in  getting  DoD  Components  to  develop  AISs  in  compliance  with 
life-cycle  management  requirements.  The  report  states  that  the  MAISRC  was 
not  aggressive  enough  in  terminating  or  redirecting  major  AIS  development 
efforts  when  the  DoD  Components  did  not  comply  with  life-cycle  management 
policies.  The  report  recommends  that  the  Secretary  of  Defense  direct  that, 
when  MAISRC  review  results  warrant,  the  MAISRC  deny  milestone  approval, 
prohibit  further  development,  and  ensure  MAISRC  decisions  are  reflected  in 
program  budgets.  The  report  also  recommends  that  the  Secretary  of  Defense 
direct  the  MAISRC  to  periodically  assess  the  adequacy  of  the  Components' 
oversight  processes  and  to  recommend  corrective  actions,  as  appropriate.  DoD 
agreed  with  the  findings  in  principle,  but  provided  no  specific  actions  to 
implement  the  recommendations. 
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Oversight  Process  for  Major  Automated 
Information  Systems 

Although  DoD  has  changed  its  approach  in  acquiring  and  developing 
AISs,  the  MAISRC  oversight  process  has  not  been  restructured  to 
effectively  incorporate  new  AIS  acquisition  and  development  methods 
and  to  consider  the  related  Corporate  Information  Management 
Initiative.  The  MAISRC: 

o  has  not  developed  procedures  to  review  major  automated 
information  systems  as  part  of  functional  process  improvements  and 
efforts  to  standardize  automated  information  systems, 

o  has  not  adequately  monitored  and  validated  major  AIS  costs 
and  benefits, 

o  has  not  developed  guidance  and  procedures  for  the  review  of 
major  AISs  that  are  incrementally  developed,  and 

o  has  not  clearly  defined  key  information  or  the  minimal 
documentation  required  for  MAISRC  reviews. 

Additionally,  the  Assistant  Secretary  of  Defense  (Command,  Control, 
Communications,  and  Intelligence)  has  not  clearly  established  oversight 
responsibility  for  major  AISs  for  command  and  control, 
communications,  and  intelligence.  Improvements  in  the  MAISRC 
process  will  provide  better  assurance  that  the  expenditure  of  about 
$3  billion  for  major  AISs  during  FYs  1995  and  1996  will  be  in  support 
of  DoD  information  management  goals  and  achieve  the  inherent 
economies  and  efficiencies  of  those  goals. 


Major  AIS  Reviews 


Development  Guidance  and  Requirements.  Historically,  DoD  has 
experienced  cost,  schedule,  and  performance  difficulties  in  the  acquisition  and 
development  of  AISs.  To  help  eliminate  those  difficulties,  DoD  instituted  a 
structured  life-cycle  management  approach  that  requires  formal  MAISRC 
milestone  reviews  for  each  life-cycle  phase  of  a  major  AIS.  DoD 
Directive  8120.1,  "Life-Cycle  Management  (LCM)  of  Automated  Information 
Systems  (AISs),"  January  14,  1993,  requires  that  AISs  be  developed  in  the  most 
cost-effective  manner  and  that  expenditures  be  controlled  "to  ensure  that  derived 
benefits  satisfy  mission  needs  .  .  .  ."  DoD  Instruction  8120.2,  "Automated 
Information  System  (AIS)  Life-Cycle  Management  (LCM)  Process,  Review, 
and  Milestone  Approval  Procedures,"  January  14,  1993,  designates  the 
responsibilities  of  the  MAISRC  and  establishes  policy  for  the  review  and 
milestone  approval  of  AISs.  DoD  Manual  7920.2-M,  "Automated  Information 
System  Life  Cycle  Management  Manual,"  March  1990,  provides  life-cycle 
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management  review  procedures  and  documentation  guidelines.  However,  those 
regulations  need  to  be  revised  to  ensure  the  MAISRC  process  incorporates  the 
goals  of  the  Corporate  Information  Management  Initiative. 

MAISRC  Oversight  Reviews.  Once  a  major  AIS  has  been  identified  for 
MAISRC  oversight,  the  MAISRC  monitors  the  status  of  the  AIS  programs  and 
provides  direction  to  the  AIS  program  manager  through  each  life-cycle  phase. 
In  accordance  with  provisions  of  DoD  Instruction  8120.2,  the  MAISRC 
delegated  15  of  the  53  major  AISs  listed  in  Appendix  C  to  DoD  Components  for 
oversight.  Our  audit  focused  on  the  MAISRC  oversight  process,  not  the 
oversight  processes  used  by  the  Component  councils.  The  day-to-day  activities 
in  the  MAISRC  process  are  performed  by  the  MAISRC  staff.  The  MAISRC 
staff  consists  of  the  action  officers  within  the  offices  of  the  MAISRC  members. 
Appendix  B  provides  more  details  on  the  MAISRC  process,  members,  and  staff. 

Types  of  Major  AIS  Reviews.  To  fulfill  its  oversight  responsibilities, 
the  MAISRC  performs  two  types  of  life-cycle  management  reviews,  a  milestone 
review  and  an  in-process  review.  A  milestone  and  an  in-process  review  are 
formal  life-cycle  management  reviews  as  prescribed  by  DoD  Instruction  8120.2. 
Those  reviews  are  also  performed  by  the  Component  review  councils.  A 
MAISRC  milestone  or  an  in-process  review  begins  with  a  "paper"  review  by  the 
MAISRC  staff.  If  the  MAISRC  staff  identifies  no  unresolved  problems,  then 
the  results  of  the  review  are  presented  to  the  MAISRC  members  in  a  draft 
System  Decision  Memorandum.  The  MAISRC  members  review  the  "paper" 
review  results  and  do  not  perform  any  additional  review.  Each  type  of  review 
may  result  in  life-cycle  management  decisions  and  direction  to  the  program 
manager  as  approved  by  the  Milestone  Decision  Authority.  Table  1  shows  the 
number  of  each  type  of  review  performed  by  the  MAISRC  and  the  DoD 
Component  councils  since  January  1993.  Of  the  19  reviews  performed  by  the 
MAISRC  since  January  1993,  5  were  paper  reviews. 


Table  1.  Completed  Reviews  Since  January  1993 


Organization 

Milestone 

In-Process 

Total 

MAISRC 

6 

13 

19 

Component  council 

_4 

_4 

_8 

Total 

10 

17 

27 

Comparison  of  Milestone  and  In-Process  Reviews.  The  basic 
differences  between  an  in-process  review  and  a  milestone  review  are  timing  and 
purpose  of  the  review  and  documentation  requirements.  A  milestone  review 
occurs  when  a  decision  is  needed  on  whether  the  major  AIS  continues  to  the 
next  milestone  or  life-cycle  management  phase.  Whereas  an  in-process  review 
may  occur  at  any  time  within  the  life-cycle  of  the  AIS.  The  in-process  review 
determines  the  program's  status,  progress  since  the  last  review,  program  risk 
and  risk-reduction  measures,  and  potential  problems  that  require  oversight 
guidance.  The  documentation  requirements  for  the  milestone  review  are 
specified  in  DoD  Manual  7920. 2-M.  The  documentation  requirements  for  an 
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in-process  review  are  based  on  the  specific  purpose  of  the  review.  For 
example,  operational  testing  may  be  the  focus  of  an  in-process  review,  and  an 
updated  test  and  evaluation  master  plan  may  be  the  only  documentation  needed. 


Supporting  Functional  Process  Improvement  Efforts 


The  MAISRC  oversight  process  does  not  effectively  support  ongoing  DoD 
efforts  to  achieve  economies  through  functional  process  improvements  and 
associated  migratory  AISs.  To  meet  statutory  and  regulatory  requirements, 
MAISRC  reviews  focus  on  compliance  with  AIS  acquisition,  development,  and 
approval  processes.  The  current  MAISRC  review  process  of  reviewing 
individual  major  AISs  one  system  at  a  time  is  representative  of  DoD's  former 
approach  to  AIS  acquisition  and  development  and  is  contrary  to  current  DoD 
information  management  initiatives.  The  individual  system  review  process 
lunits  the  MAISRC  staff's  awareness  of  the  status  and  implementation  of 
functional  process  improvements  and  migratory  AISs,  thereby  affecting  the 
overall  value  and  effectiveness  of  MAISRC  reviews. 

Increasing  MAISRC  Awareness  of  Functional  Improvements.  Because  the 
MAISRC  reviews  major  AISs  one  system  at  a  time,  the  MAISRC  staff  is  not 
well  informed  of  other  complementary  systems  and  efforts  that  improve 
functional  processes.  If  the  MAISRC  staff  was  more  involved  in  efforts  related 
to  the  improvement  of  functional  systems  and  other  aspects  of  the  Corporate 
Information  Management  Initiative,  the  staff  would  be  able  to  perform  a  more 
encompassing  review.  That  involvement  would  help  provide  the  staff  answers 
to  crucial  AIS  review  questions,  such  as  the  following. 

o  Is  the  AIS  being  developed  to  implement  an  improved  functional 
process? 

o  If  the  AIS  program  is  not  a  migration  system  within  its  functional 
area,  will  it  serve  a  duplicative  purpose?  If  the  AIS  is  a  Imigration  system,  how 
and  when  will  the  legacy  systems  be  terminated? 

o  Are  the  hardware  and  software  based  on  an  approved  architecture,  and 
do  they  incorporate  appropriate  standards  and  use  appropriate  development 
methodologies  and  concepts? 


*An  existing  or  planned  and  approved  AIS  that  has  been  designated  to  support  a 
functional  process  on  a  DoD-wide  basis. 
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o  Does  the  program  adequately  consider  using  the  evolving  information 
infrastructure  for  related  communications  and  computer-processing 
requirements? 

o  Has  the  program  used  the  standardized  data  definitions  developed  for 
DoD-wide  use? 

Involvement  in  ongoing  DoD  Corporate  Information  Management  Initiative 
efforts  would  enable  the  MAISRC  staff  to  perform,  from  a  DoD  perspective,  a 
more  effective  AIS  review,  and  from  the  standpoint  of  the  responsible  Principal 
Staff  Assistant,  a  more  valuable  review. 

Concurrent  Involvement  with  Migratory  AISs.  DoD  Principal  Staff 
Assistants  are  identifying  migratory  AISs  to  achieve  economies  and  efficiencies 
in  their  respective  functions.  About  165  migratory  AISs  have  been  identified 
for  9  functional  areas.  Those  AISs  include  15  systems  subject  to  MAISRC 
review.  The  critical  factors  of  costs  and  benefits  and  the  successful  deployment 
of  efficient  and  effective  systems  are  of  interest  to  both  the  Principal  Staff 
Assistants  and  the  MAISRC.  However,  the  MAISRC  review  process  is  not  an 
integrated  part  of  the  ongoing  identification  and  development  of  AISs  that  will 
be  used  on  a  DoD-wide  basis.  By  being  actively  involved  in  the  identification 
and  development  of  migratory  AISs,  the  MAISRC  could  provide  an  independent 
assessment  of  critical  factors  needed  to  verify  that  the  development  of  specific 
major  AISs  will  achieve  desired  economies  and  efficiencies. 


AIS  Costs  and  Benefits 


The  MAISRC  did  not  adequately  monitor  or  evaluate  AIS  costs  and  benefits. 
As  of  January  1994,  52  AISs  were  subject  to  MAISRC  oversight.  However, 
returns  on  investment  (derived  from  costs  and  benefits)  were  not  routinely 
tracked  by  the  MAISRC.  The  return  on  investment  was  available  for  only  1  of 
those  52  AISs.  The  MAISRC  staff  informed  us  that,  as  of  December  1994, 
return  on  investment  analyses  are  required  in  the  quarterly  reports  from  the 
program  managers  of  major  AISs.  Also,  AIS  costs  and  benefits  were  not 
routinely  evaluated  or  validated  during  formal  MAISRC  reviews. 

Validating  AIS  Costs  and  Benefits.  The  Director,  Program  Analysis  and 
Evaluation,  did  not  validate  costs  and  benefits  for  12  of  the  19  major  AISs 
reviewed  by  the  MAISRC  during  1993  and  1994.  DoD  Directive  8120.1 
requires  that  costs  and  benefits  be  validated  during  the  MAISRC  process.  Costs 
and  benefit  validations  were  usually  performed  during  milestone  reviews,  but 
were  not  usually  performed  during  in-process  reviews.  Because  in-process 
reviews  have  no  standard  requirements,  AIS  program  costs  and  benefits  may  not 
be  a  focus,  and  the  Director,  Program  Analysis  and  Evaluation,  may  not  be 
materially  involved  during  the  in-process  review.  Table  2  summarizes  the  AIS 
cost  validation  efforts  during  formal  MAISRC  reviews  held  in  1993  and  1994. 


11 


Oversight  Process  for  Major  Automated  Information  Systems 


Table  2.  Summary  of  Major  AIS  Cost  Validations 
by  Director,  Program  Analysis  and  Evaluation 


Review 

Tvoe 

Incomplete 

Cost 

Data 

No 

Cost 

Data 

No 

Material 

Involvement 

Costs 

Validated 

Totals 

Milestone 

i 

1* 

0 

4 

6 

In  Process 

5 

2 

3 

3 

13 

Total 

6 

3 

3 

7 

19 

s|c 

Cost  data  not  required  at  Milestone  0 


Reasons  for  Cost-Benefits  Analysis.  The  analysis  and  validation  of  major  AIS 
costs  and  benefits  is  critical  in  evaluating  AIS  program  management  and  in 
determining  the  program's  cost-effectiveness.  The  purpose  of  a  cost-benefits 
analysis  is  to  explore  and  quantify  the  relative  cost  advantages  of  concept  and 
design  options.  A  fundamental  AIS  principle  is  that,  in  addition  to  being 
affordable,  the  most  advantageous  AIS  alternative  available  to  the  Government 
should  satisfy  all  critical  mission  requirements  at  the  lowest  life-cycle  cost.  The 
Director,  Program  Analysis  and  Evaluation,  uses  the  "AIS  Benefit/Cost  Excel 
4.0a  Model,"  June  1994,  in  validating  costs  and  benefits.  The  "Program 
Analysis  and  Evaluation/Planning  and  Analytic  Support  User  Manual"  for  that 
model  states  that  "From  a  purely  economic  perspective,  AlS  acquisition 
programs  should  support  at  least  a  10  percent  return  on  investment .  .  .  . " 

Congressional  Concerns  on  AIS  Investments.  Members  of  Congress  are 
concerned  about  DoD's  return  on  investment  for  information  systems.  In  its 
June  4,  1993,  letter  to  the  Comptroller  General,  the  Subcommittee  on 
Readiness,  House  Committee  on  Armed  Services  (now  the  Subcommittee  on 
Readiness,  House  Committee  on  National  Security)  stated  that  over  time, 
projected  benefits  were  decreasing  for  DoD  planned  investments,  totaling  about 
$80  billion,  in  information  systems  and  resources.  The  letter  states  "We  fear 
there  is  no  end-game  and  that  scarce  DoD  resources  will  be  spent  with  no 
immediate  and  little  long-term  gain."  Consistent  validation  of  costs  and 
benefits  would  strengthen  the  effectiveness  of  the  MAISRC  process  and  would 
help  assure  members  of  Congress  that  DoD  is  concerned  about  the 
cost-effectiveness  of  its  information  systems  and  management  practices. 


Oversight  of  AISs  Developed  in  Increments 


In  an  effort  to  reduce  the  time  typically  needed  to  acquire  and  develop  a  major 
AIS,  DoD  has  begun  to  develop  migratory  systems  in  a  "buildings  blocks" 
approach.  Instead  of  developing  the  entire  system  at  one  time,  the  system  is 
developed  in  increments,  or  blocks,  of  capability.  That  approach  should  get 
automated  capability  into  the  hands  of  the  users  more  quickly.  However,  the 
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building  blocks  approach  raises  new  problems  for  AIS  program  management 
and  oversight,  especially  when  requirements  for  each  increment  are  not  initially 
well  defined. 

Milestone  Approval  Process.  Incremental  development  will  be  difficult  to 
oversee  using  the  traditional  milestone  approval  process.  Because  existing  AIS 
life-cycle  management  policy  for  MAISRC  systems  does  not  provide  for 
consistent  procedures  to  review  systems  developed  incrementally,  oversight 
officials  are  evaluating  a  "moving  target"  and  will  find  traditional  cost, 
schedule,  and  performance  standards  difficult  to  apply.  DoD  life-cycle 
management  policies  need  to  be  revised  to  provide  procedures  for  the  MAISRC 
to  effectively  oversee  the  management  of  incrementally  developed  AISs. 

Operational  Testing  for  and  Validating  Costs  of  Incremental  AISs.  Existing 
acquisition  and  life-cycle  management  requirements  and  procedures  are  not 
easily  applied  to  AISs  developed  in  increments,  especially  those  concerning 
operational  testing  and  cost  validation.  Operational  test  and  evaluation,  required 
by  AIS  acquisition  policy,  verifies  that  an  AIS  performs  as  intended  and  is 
ready  to  be  deployed.  The  Director,  Operational  Test  and  Evaluation,  is  a 
member  of  the  MAISRC.  Operational  testing  procedures,  premised  on 
assessing  completed  weapon  systems  developed  by  major  acquisition  programs, 
are  lengthy  and  costly  when  performed  several  times  during  the  development  of 
an  incremental  AIS.  Additionally,  the  question  of  what  constitutes  appropriate 
operational  test  and  evaluation  for  migration  AISs  which  involve  the  incremental 
integration  of  multiple  AISs,  at  varied  life-cycle  phases,  is  a  challenging  issue 
for  the  MAISRC. 

DoD  Instruction  8120.2  requires  that  the  validation  of  costs  and  benefits  for 
major  AISs  be  performed  at  AIS  life-cycle  milestones.  The  MAISRC  has  not 
resolved  the  issue  of  how  to  apply  milestone  review  requirements  to  the  review 
of  AISs  developed  in  increments;  therefore  the  MAISRC  is  not  validating  costs 
and  benefits  for  each  increment  of  an  major  AIS.  For  effective  MAISRC 
oversight  of  AISs  developed  in  increments,  DoD  regulations  should  specify  the 
life-cycle  management  requirements  for  operational  testing  and  evaluation  and 
the  criteria  for  validating  costs  and  benefits  of  those  incremental  systems. 


Documentation  for  MAISRC  Reviews 


Documentation  Requirements.  DoD  guidance  does  not  define  minimum 
documentation  requirements  for  in-process  reviews.  The  DoD 
Manual  7920. 2-M  specifies  the  documentation  needed  for  milestone  reviews, 
and  DoD  Instruction  8120.2  only  describes  the  nature  of  the  documentation  for 
in-process  reviews.  The  program  manager  must  satisfactorily  complete  needed 
documentation  before  each  milestone  review.  The  documentation  supports  the 
program,  functional,  and  technical  concepts  that  are  determined  from  the 
planning  and  analysis  tasks  completed  by  the  program  managers. 
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Documentation  Effects  on  Reviews.  The  MAISRC  staff  considers  the 
documentation  supporting  a  milestone  review  a  significant  burden  to  review, 
and  program  managers  consider  the  documentation  a  significant  burden  to 
prepare.  Difficulties  in  preparing  documentation  for  milestone  reviews  are 
compounded  by  Component  review  councils'  requests  for  more  information. 
Consequently,  we  believe  the  MAISRC  has  performed  more  frequent  in-process 
reviews  to  obtain  critical  information  on  the  progress  of  major  AISs  and  fewer 
milestone  reviews.  Of  the  27  reviews  performed  since  January  1993, 
17  (63  percent)  were  in-process  reviews  (see  Appendix  B). 

Key  Management  Information  Needs.  The  MAISRC  has  not  established  the 
key  management  information  requirements  for  MAISRC  oversight  reviews. 
Key  management  information,  such  as  validated  costs  and  benefits,  was  not 
consistently  provided  for  each  review  performed  since  January  1993,  (see 
Table  2).  To  alleviate  the  burden  of  preparing  documentation,  key  information 
requirements  should  be  designated  for  milestone  and  in-process  reviews. 


Oversight  of  Command  and  Control,  Communications,  and 
Intelligence  Systems 


DoD  guidance  does  not  explicitly  delineate  responsibility  for  oversight  of 
command  and  control,  communications,  and  intelligence  AISs.  The  Office  of 
the  Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence)  has  estimated  that  thousands  of  command  and  control, 
communications,  and  intelligence  AISs  exist.  The  investment  and  annual  costs 
of  those  AISs  are  generally  thought  to  be  substantial,  but  costs  are  not  well 
defined  because  they  are  not  separately  accounted  for  in  the  budgeting  and 
appropriations  process.  Accordingly,  command  and  control,  communications, 
and  intelligence  AISs  do  not  receive  the  same  type  of  oversight  given  to 
"business  or  administrative"  AISs,  for  which  detailed  budgetary  information  is 
required  in  DoD's  annual  budget  submission  to  Congress. 

DoD  Guidance  Related  to  Command  and  Control,  Communications,  and 
Intelligence  Systems.  Though  not  required  by  law,  DoD  has  traditionally 
separated  the  application  of  DoD  guidance  for  the  oversight  of  AISs,  depending 
on  the  purpose  for  each  system.  AISs  deemed  to  be  critical  to  the  mission 
performance  of  weapon  systems  have  been  acquired  and  managed  in  accordance 
with  requirements  primarily  in  DoD  Directive  5000.1,  "Defense  Acquisition," 
February  23,  1991,  and  DoD  Instruction  5000.2,  "Defense  Acquisition 
Management  Policy  and  Procedures,"  February  26,  1993.  Those 

mission-critical  systems  that  meet  specific  monetary  thresholds  are  subject  to 
oversight  by  the  Defense  Acquisition  Board,  not  the  MAISRC.  The  scope  of 
DoD  Directive  8000.1,  "Defense  Information  Management  (IM)  Program," 
October  27,  1992,  specifically  includes  "information  management  resources  and 
services  used  for  ...  command,  control,  communications,  and  intelligence 
unless  specifically  exempted  by  the  Assistant  Secretary  of  Defense  (Command, 
Control,  Communications,  and  Intelligence)."  We  identified  no  command  and 
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control,  communications,  and  intelligence  AISs  that  had  been  exempted  by  the 
Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence). 

Section  D  ("Computer  Resources"),  part  6,  of  DoD  Instruction  5000.2  provides 
procedures  that  apply  to  "computer  resources,  hardware  and  software  that  are 
physically  part  of,  dedicated  to,  or  essential  in  real  time  to  the  mission 
performance  of  weapon  systems."  However,  the  Instruction  does  not  specify 
what  is  "essential  in  real  time  to  the  mission  performance  of  weapon  systems. " 
Accordingly,  program  management  decisions  to  apply  DoD  Instruction  5000.2 
for  command  and  control,  communications,  and  intelligence  AISs  are,  in  our 
opinion,  highly  subjective. 

Defense  Acquisition  Board  Oversight  Versus  MAISRC  Oversight.  The 
monetary  thresholds  for  determining  oversight  by  the  MAISRC  or  the  Defense 
Acquisition  Board  differ  significantly.  The  threshold  for  a  major  AIS  subject  to 
MAISRC  oversight  is  $300  million  in  life-cycle  costs,  which  includes  costs  of 
all  types,  regardless  of  how  they  are  funded.  On  the  other  hand,  the  primary 
Defense  Acquisition  Board  threshold  is  $1.8  billion  in  procurement  binding. 
Defense  Acquisition  Board  thresholds  do  not  include  operation  and  maintenance 
funding,  which  comprises  a  large  part  of  life-cycle  costs. 

We  concluded  that  guidance  in  DoD  Directive  8000.1  is  applicable  to  most 
command  and  control,  communications,  and  intelligence  major  AISs.  Because 
of  the  difference  in  qualifying  thresholds,  program  managers  have  generally 
applied  guidance  in  DoD  Instruction  5000.2  to  command  and  control, 
communications,  and  intelligence  systems,  thereby  avoiding  the  MAISRC s 
lower  oversight  thresholds.  However,  the  MAISRC  has  reviewed 
eight  command  and  control  systems.  We  believe  that  other  command  and 
control,  communications,  and  intelligence  AISs  meet  the  thresholds  for  and 
should  be  provided  oversight  by  the  MAISRC. 


Ongoing  Efforts  to  Improve  the  MAISRC  Process 


The  Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence)  has  initiated  efforts  to  examine  and  improve  the  structure  and 
operations  of  his  office  and  that  of  the  MAISRC.  Personnel  from  the  Office  of 
the  Assistant  Secretary  took  part  in  the  Secretary  of  Defense  Process  Action 
Team  for  Acquisition  Reform.  The  process  action  team's  conclusions  and 
recommendations  are  summarized  under  Prior  Audits  and  Other  Reviews  in 
Parti  of  this  report.  The  Assistant  Secretary  also  authorized  two  functional 
process  improvement  studies.  One  study  directly  focuses  on  the  MAISRC 
process,  and  the  other  considers  MAISRC  as  one  of  the  functions  of  the  Office 
of  the  Assistant  Secretary.  The  thrust  and  status  of  those  functional  process 
improvement  studies  are  described  below. 

MAISRC  Functional  Process  Improvement  Study.  The  objective  of  the 
MAISRC  Functional  Process  Improvement  study  is  to  improve  the  MAISRC 
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process  and  data  and  streamline  oversight  of  the  life-cycle  management  process 
for  the  development  and  operation  of  major  AISs  under  MAISRC  oversight.  In 
November  1994,  an  interim  report  was  issued,  presenting  a  model  of  the  current 
MAISRC  process  as  viewed  by  the  MAISRC  staff.  In  March  1995,  the  study 
team  issued  a  report  which  modeled  the  MAISRC  process  as  viewed  by  major 
AIS  program  managers.  The  study  team  will  meld  the  two  perspectives  into  an 
overall  model  of  the  MAISRC  process  and  will  identify  the  changes  needed  for 
an  improved  MAISRC  process.  Expected  study  outcomes  include  changes  to 
DoD  Directive  8120.1  and  DoD  Instruction  8120.2  and  revision  of  DoD 
Manual  7920. 2-M,  which  is  planned  to  be  reissued  as  DoD  Manual  8120. 2-M, 
"Automated  Information  Systems,  Life-Cycle  Management  Manual." 

Enterprise  Analysis  for  the  Assistant  Secretary.  The  Enterprise  Analysis  for 
the  Assistant  Secretary  study  is  designed  to  improve  the  effectiveness  and 
efficiency  of  the  mission  performance  of  the  Office  of  the  Assistant  Secretary  of 
Defense  (Command,  Control,  Communications,  and  Intelligence).  A  draft 
report  issued  September  23,  1994,  includes  recommendations  designed  to 
improve  the  Office  of  the  Assistant  Secretary's  structure,  process,  and 
environment.  The  Assistant  Secretary's  staff  is  evaluating  the 
recommendations.  Implementation  of  the  recommendations  will  be  the  decision 
of  the  Assistant  Secretary.  No  recommendations  specifically  apply  to  the 
MAISRC  process. 


Summary 


Improvements  to  the  MAISRC  process  will  better  support  the  goals  of  the  DoD 
Corporate  Information  Management  Initiative.  Also,  improvements  in  the 
process  will  provide  an  independent  assessment  of  critical  factors  needed  to 
verify  that  major  AISs  under  development  will  achieve  the  desired  economies 
and  efficiencies. 


Recommendations,  Management  Comments,  and  Audit 
Response 


Revised  and  Renumbered  Recommendations.  Because  procedures  involving 
the  MAISRC  staff  in  Corporate  Information  Management  Initiative  efforts  do 
not  need  to  be  specified  in  DoD  AIS  life-cycle  management  directives  and 
guidance,  we  renumbered  draft  Recommendation  1.  as  Recommendation  3.  As 
a  result  of  management  comments,  we  revised  draft  Recommendations  4.  and  5. 
(renumbered  as  Recommendation  6.)  to  clarify  our  intent.  Also,  we 
renumbered  draft  Recommendations  2.  and  3.  to  1.  and  2.,  respectively,  and  to 
better  respond  to  management  comments,  we  renumbered  draft 
Recommendations  5.  and  6.  to  6.  and  5.,  respectively. 
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We  recommend  that  the  Assistant  Secretary  of  Defense  (Command, 
Control,  Communications,  and  Intelligence): 

1.  Specify  procedures  that  will  involve  the  Major  Automated  Information 
Systems  Review  Council  staff  in  the  ongoing  efforts  of  the  DoD  Corporate 
Information  Management  Initiative. 

2.  Specify  procedures  for  the  Major  Automated  Information  System 
Review  Council  staff  to  perform  assessments  of,  and  concurrent  with,  the 
identification  and  development  of  each  migratory  major  automated 
information  system  that  will  be  used  on  a  DoD-wide  basis. 

Management  Comments.  The  Deputy  Assistant  Secretary  (Acquisition), 
responding  for  the  Assistant  Secretary  of  Defense  (Command,  Control, 
Communications  and  Intelligence),  nonconcurred  with  both  recommendations. 
In  response  to  Recommendation  1.,  the  Deputy  Assistant  Secretary  indicated 
that  the  MAISRC  staff  already  is  involved  with  the  DoD  Corporate  Information 
Management  Initiative  in  that  the  staff  reviews  any  major  automated  information 
system  that  is  evolving  in  response  to  the  Corporate  Information  Management 
Initiative.  Regarding  Recommendation  2.,  the  function  of  the  MAISRC  is 
oversight,  and  any  MAISRC  involvement  would  be  inappropriate  during  the 
identification  of  candidates  for  and  selection  of  migratory  AISs.  The 
designation  of  migratory  AISs  is  a  responsibility  of  the  Principal  Staff 
Assistants,  with  evaluation  criteria  provided  by  the  Office  of  the  Assistant 
Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence) 
and  migration  assessment  support  provided  by  the  Defense  Information  Systems 
Agency.  The  staff  of  the  Assistant  Secretary  also  reviews  and  approves  each 
proposed  migratory  system.  About  half  of  the  major  AISs  overseen  by  the 
MAISRC  are  migratory  systems. 

Audit  Response.  We  do  not  consider  the  Deputy  Assistant  Secretary's 
comments  responsive.  The  MAISRC  oversight  function  can  best  be  performed 
by  a  staff  with  a  broad  understanding  and  awareness  of  DoD  information 
management  initiatives,  including  the  related  migration  activities  within  DoD's 
functional  areas.  Although  statutory  compliance  must  remain  a  concern  of  the 
MAISRC,  the  MAISRC 's  review  approach  and  focus  must  change  to  effectively 
correspond  to  the  DoD  "corporate"  perspective.  Active  involvement  in 
Corporate  Information  Management  Initiative  efforts  would  help  provide  the 
MAISRC  staff  the  overall  knowledge  and  insight  needed  to  review  AISs  from 
that  corporate  perspective.  The  adoption  of  the  Integrated  Product  Team 
approach  for  the  oversight  of  weapon  system  acquisition,  which  is  a  key  element 
of  ongoing  DoD  acquisition  reform,  may  well  serve  as  a  useful  model  for  the 
AIS  area. 

Active  participation  by  the  MAISRC  staff  in  Corporate  Information 
Management  Initiative  efforts,  including  the  identification  and  selection  of 
migratory  AISs,  would  not  in  any  way  compromise  the  independence  required 
for  the  MAISRC  review  function.  The  knowledge  gained  from  active 
involvement  in  Corporate  Information  Management  Initiative  efforts  would 
provide  a  clearer  "big  picture"  of  DoD  information  management  and  enable 
more  thorough  and  encompassing  MAISRC  reviews.  Concurrent  migration 
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system  involvement  would  provide  the  opportunity  for  the  MAISRC  staff, 
Principal  Staff  Assistants,  and  major  AIS  program  managers  to  become  better 
aware  of  each  other's  views  and  perspectives  and  to  discuss  related  issues  before 
those  issues  become  problems.  Collaborative  efforts  should  result  in  major 
AISs  that  provide  effective  support  and  achieve  the  economies  and  efficiencies 
sought.  We  ask  that  management  provide  additional  comments  on  the 
recommendations. 

3.  Revise  DoD  Directive  8120.1,  "Life-Cycle  Management  of  Automated 

Information  Systems,"  January  14,  1993;  DoD  Instruction  8120.2, 

"Automated  Information  System  Life-Cycle  Management  Process,  Review, 
and  Milestone  Approval  Procedures,"  January  14,  1993;  and  DoD  Manual 
7920.2-M,  "Automated  Information  Systems,  Life-Cycle  Management 
Manual,"  March  1990,  to  reflect  the  provisions  in 
Recommendations  4.  through  8. 

Management  Comments.  As  described  under  the  following  recommendations, 
the  Deputy  Assistant  Secretary  generally  concurred  and  stated  that  revisions  to 
DoD  Manual  7920.2-M  will  resolve  most  of  the  issues.  He  also  indicated  that, 
as  a  result  of  ongoing  process  improvement  initiatives,  DoD  Directive  8120.1 
and  DoD  Instruction  8120.2  would  also  be  revised  to  achieve  other 
recommended  actions. 

Audit  Response.  The  Deputy  Assistant  Secretary's  comments  as  applicable  to 
Recommendations  5.  and  7.  are  responsive.  We  request  that  the  Assistant 
Secretary  provide  additional  comments  on  Recommendations  4.,  6.,  and  8.  as 
discussed  below. 

4.  Designate  the  Director,  Program  Analysis  and  Evaluation,  to  validate 
the  costs  and  benefits  of  a  major  automated  information  system  at 
established  points  throughout  its  development. 

Management  Comments.  The  Deputy  Assistant  Secretary  concurred,  stating 
that  DoD  Directive  8120.1  and  DoD  Instruction  8120.2  presently  task  the 
Director,  Program  Analysis  and  Evaluation,  to  review  and  validate  the  costs  and 
benefits  of  a  major  AIS  throughout  its  life  cycle.  Also,  clarifying  guidance  will 
be  issued  by  the  Office  of  the  Director,  Program  Analysis  and  Evaluation,  and 
in  DoD  Manual  8120.2-M. 

Audit  Response.  The  Deputy  Assistant  Secretary's  comments  are  partially 
responsive.  The  intent  of  our  recommendation  is  much  broader  than  the  Deputy 
Assistant  Secretary's  interpretation.  We  agree  that  the  Director,  Program 
Analysis  and  Evaluation,  is  responsible  for  the  review  and  validation  of  the  cost 
and  benefits  of  major  AIS  programs  at  appropriate  life-cycle  management 
reviews;  however,  we  believe  that  costs  and  benefits  are  not  adequately 
validated.  In-process  reviews  have  become  the  predominant  type  of  MAISRC 
review.  However,  as  detailed  in  Table  2  (page  12),  the  Director,  Program 
Analysis  and  Evaluation,  did  not  validate  the  costs  and  benefits  for  10  of  the  13 
in-process  reviews  performed  by  the  MAISRC  during  1993  and  1994.  Cost  and 
benefit  validations  are  required  for  most  milestone  reviews,  but  in-process 
review  requirements  are  not  uniform  and  the  Director,  Program  Analysis  and 
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Evaluation,  may  have  no  in-process  review  validation  responsibilities.  We 
believe  that  periodic  cost  and  benefit  validations,  scheduled  by  the  MAISRC 
without  regard  to  type  of  oversight  review,  would  help  ensure  positive  returns 
on  investments  and  would  illustrate  the  DoD  resolve  to  spend  its  limited 
resources  wisely.  We  request  that  management  provide  additional  comments  on 
the  recommendation. 

5.  Specify  life-cycle  management  requirements  for  validating  costs  and 
benefits  and  for  performing  operational  tests  and  evaluations  of  major 
automated  information  systems  developed  in  increments.  Determine  the 
life-cycle  management  requirements  in  coordination  with  the  Director, 
Program  Analysis  and  Evaluation,  and  the  Director,  Operational  Test  and 
Evaluation. 

Management  Comments.  The  Deputy  Assistant  Secretary  concurred.  In 
response  to  validating  costs  and  benefits  of  AISs  developed  in  increments,  the 
Deputy  Assistant  Secretary  stated  that  DoD  Instruction  8120.2  provides  for 
combined  or  repeated  milestone  decision  points  for  the  oversight  of  incremental 
AIS  programs.  He  recognized  that  the  costs  and  benefits  of  incremental 
programs  are  difficult  to  estimate  because  different  increments  are  at  various 
stages  of  definition,  development,  and  deployment.  Costs  and  benefits  are 
usually  more  accurate  and  verifiable  for  increments  that  are  closest  to 
deployment.  The  hardware  and  system  configuration  investment  required  for 
early  increments  may  distort  a  program's  return  on  investment  during  its  initial 
stages.  The  Director,  Program  Analysis  and  Evaluation,  had  considered  those 
factors  in  developing  draft  AIS  Economic  Analysis  procedures  for  inclusion  in 
DoD  Manual  8120. 2-M,  which  will  be  issued  in  September  1995.  The 
Director,  Program  Analysis  and  Evaluation,  also  plans  to  issue  a  revised  "DoD 
Automated  Information  Systems  (AIS)  Economic  Analysis  (EA)  Guide"  in 
October  1995. 

In  response  to  operational  tests  and  evaluations  of  major  AISs  developed  in 
increments,  the  Director,  Operational  Test  and  Evaluation,  presently  oversees 
test  and  evaluation  planning  and  execution  of  major  AISs,  including  those 
developed  in  increments.  The  Deputy  Assistant  Secretary  stated  that  the  report 
incorrectly  states  that  operational  testing  is  not  performed  on  AISs  developed  in 
increments.  Specific  procedures  for  performing  operational  testing  of 
incremental  AIS  programs  are  being  developed  and  will  be  included  in  the 
pending  DoD  Manual  8120.2. 

Audit  Response.  The  Deputy  Assistant  Secretary's  comments  meet  the  intent 
of  the  recommendation,  and  no  further  response  is  required.  Regarding  the 
operational  testing  of  AISs  developed  in  increments,  we  agree  that  the  draft 
report  discussion  could  be  misleading,  and  we  revised  that  discussion  to  clarify 
our  intent. 

6.  Establish  procedures  and  criteria  to  oversee  the  cost,  schedule,  and 
performance  of  incrementally  developed  major  automated  information 
systems  for  which  firm  requirements  are  not  initially  defined. 
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Management  Comments.  The  Deputy  Assistant  Secretary  concurred  that 
life-cycle  management  directives  should  include  procedures  and  criteria  for 
overseeing  incrementally  developed  AISs.  The  AIS  program  managers  are 
required  to  define  and  obtain  MAISRC  approval  of  a  tailored  strategy  for 
program  management.  The  Deputy  Assistant  Secretary  stated  that  he  would 
consider  clarifying  procedures  for  the  oversight  of  incrementally  developed 
AISs  during  the  revision  of  DoD  Directive  8120.1  and  DoD  Instruction  8120.2. 
Those  revisions  are  scheduled  to  begin  in  July  1995. 

Audit  Response.  The  Deputy  Assistant  Secretary's  comments  are  not 
responsive.  We  agree  that  the  present  life-cycle  guidance  provides  for  tailored 
strategies,  but  that  is  not  the  focus  of  our  recommendation.  Our  intent  is  to 
focus  on  the  oversight  of  the  execution  of  those  tailored  strategies,  especially  the 
strategies  involving  incremental  development  of  AISs  for  which  functional  and 
technical  requirements  are  not  initially  well  defined.  Methods  and  mechanisms 
exist  to  help  monitor  and  track  the  cost,  schedule,  and  performance  of  an  AIS 
development.  However,  most  of  those  methods  and  mechanisms  are  premised 
on  well-defined  requirements.  The  performance  of  a  traditional  cost-benefit 
analysis,  for  example,  may  provide  little  assistance  to  the  oversight  and  control 
of  the  cost  of  a  system  being  developed  to  meet  evolving  requirements. 
Accordingly,  we  believe  that  procedures  and  criteria  need  to  be  established  to 
effectively  oversee  the  incremental  development  of  major  AISs.  We  revised  the 
recommendation  to  clarify  our  intent,  and  we  request  additional  comments  on 
the  revised  recommendation. 

7.  Designate  key  program  management  information,  such  as  validated 
cost-benefits  analyses  and  operational  test  plans,  for  milestone  and 
in-process  reviews  performed  by  the  Major  Automated  Information 
Systems  Review  Council. 

Management  Comments.  The  Deputy  Assistant  Secretary  concurred,  stating 
that  improved  guidelines  for  AIS  life-cycle  management  review  documentation 
will  be  provided  in  DoD  Manual  8120. 2-M,  "Automated  Information  System 
Life  Cycle  Management  Manual, "  which  is  scheduled  to  be  issued  by  October 
1995. 

8.  Clarify  the  oversight  responsibilities  of  the  Major  Automated 
Information  Systems  Review  Council  for  command  and  control, 
communications,  and  intelligence  major  automated  information  systems 
that  have  not  been  exempted  from  DoD  Directive  8000.1,  "Defense 
Information  Management  Program,"  October  27,  1992. 

Management  Comments.  The  Deputy  Assistant  Secretary  concurred,  stating 
that  revisions  of  DoD  Directive  8120.1  and  DoD  Instruction  8120.2,  scheduled 
to  begin  in  July  1995,  will  further  clarify  that  major  AISs  used  for  command 
and  control  are  subject  to  MAISRC  oversight. 

Audit  Response.  Although  the  Deputy  Assistant  Secretary  concurred,  the 
comments  are  limited  to  command  and  control  major  AISs.  Because  the 


20 


Oversight  Process  for  Major  Automated  Information  Systems 


recommendation  also  includes  communications  and  intelligence  major  AISs,  we 
ask  that  management  provide  additional  comments,  explaining  the  planned 
actions  to  clarify  oversight  responsibilities  for  those  major  AISs. 


Management  Comments  Required 


Responses  to  the  final  report  are  required  for  the  items  indicated  with  an  "X"  in 
the  table  below. 

Table  3.  Response  Requirement  for  Each  Recommendation 


Response  Should  Cover 

Recommendation 

Reconsider 

Proposed 

Completion 

Number 

Position 

Action 

Date 

i. 

X 

X 

X 

2. 

X 

X 

X 

3. 

X 

X 

X 

4. 

N/A 

X 

X 

6. 

N/A 

x* 

X 

8. 

N/A 

X 

X 

"indicate  position  on  clarifying  oversight  responsibilities  for  communications 
and  intelligence  major  AISs. 
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Part  III  -  Additional  Information 


Appendix  A.  Major  AIS  Criteria  and  Life-Cycle 

Management  Milestone  Reviews 

Major  Automated  Information  Systems  Criteria.  Using  FY  1990  constant 
dollars,  a  major  automated  information  system: 

o  has  anticipated  program  costs  of  more  than  $100  million,  or 

o  has  estimated  program  costs  of  more  than  $25  million  in  a  single  year, 
or 


o  has  estimated  life-cycle  costs  of  more  than  $300  million,  or 

o  has  been  so  designated  by  the  milestone  decision  authority. 

Life-Cycle  Management  Milestone  Reviews.  DoD  Directive  8120.1 
establishes  life-cycle  management  requirements  for  automated  information 
systems  (AISs).  Milestone  reviews  are  held  between  life-cycle  management 
phases  to  evaluate  AIS  program  performance  during  the  preceding  phase,  assess 
plans  for  the  rest  of  the  program,  and  establish  exit  criteria*  for  die  next  phase. 
The  MAISRC  milestone  reviews  and  their  basic  purposes  follow. 

o  Milestone  0.  Verifies  the  mission  need  statement  and  authorizes  the 
concept  exploration  and  definition  phase. 

o  Milestone  I.  Selects  the  best  program  concept  alternative  and 
authorizes  the  validation  phase. 

o  Milestone  II.  Determines  whether  the  validated  system  will  meet 
mission  needs  and  authorizes  the  development  phase. 

o  Milestone  III.  Determines  whether  the  developed  system  is  ready  for 
deployment  and,  if  so,  authorizes  production  and  deployment. 

o  Milestone  IV.  Evaluates  effectiveness  of  operational  support  of  the 
AIS  and  decides  whether  to  continue,  modernize,  or  terminate  the  system. 


*Specific  program  criteria  that  must  be  accomplished  before  proceeding  to  the 
next  milestone  phase. 
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MAISRC  Members.  The  Major  Automated  Information  Systems  Review 
Council  (MAISRC)  is  composed  of  the  MAISRC  Chair,  the  MAISRC  members, 
the  MAISRC  Executive  Secretary,  and  the  MAISRC  staff.  The  Assistant 
Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence)  is 
the  chairperson  and  life-cycle  management  milestone  decision  authority.  Other 
MAISRC  members  are  the: 

o  Secretary  of  Defense  Principal  Staff  Assistants,  including  those  who 
provide  management  responsibility  for  the  functional  areas  supported  by  the 
AISs  subject  to  review; 

o  Under  Secretary  of  Defense  (Comptroller); 

o  Director,  Program  Analysis  and  Evaluation,  Office  of  the  Under 
Secretary  of  Defense  (Comptroller); 

o  Assistant  Secretary  of  Defense  (Reserve  Affairs),  when  appropriate; 

o  Director,  Acquisition  Policy  and  Program  Integration,  Office  of  the 
Under  Secretary  of  Defense  for  Acquisition  and  Technology; 

o  Director,  Test  and  Evaluation,  Office  of  the  Under  Secretary  of 
Defense  for  Acquisition  and  Technology; 

o  Director,  Operational  Test  and  Evaluation; 

o  Chairman,  Joint  Chiefs  of  Staff,  representative(s); 

o  Senior  Acquisition  Authority  for  the  AIS  program  subject  to  review; 
and 


o  other  members  selected  at  the  discretion  of  the  milestone  decision 
authority. 

The  MAISRC  Staff.  The  MAISRC  staff  is  comprised  of  action  officers  within 
the  offices  of  the  MAISRC  members.  For  example,  in  meeting  his 
responsibility  for  the  oversight  of  major  AISs,  the  Assistant  Secretary  of 
Defense  (Command,  Control,  Communications,  and  Intelligence)  relies  on 
action  officers  in  the  Office  of  the  Deputy  Assistant  Secretary  of  Defense 
(Acquisition)  for  day-to-day  support.  Other  MAISRC  staff  work  in  the  offices 
of  the  Director,  Program  Analysis  and  Evaluation;  the  Director,  Operational 
Test  and  Evaluation;  and  the  Principal  Staff  Assistant  for  the  functional  area 
supported  by  the  AIS  subject  to  review.  MAISRC  staff  in  the  Office  of  the 
Deputy  Assistant  Secretary  of  Defense  (Acquisition)  normally  coordinate  the 
oversight  activities  of  other  staff  working  in  the  offices  of  the  other  MAISRC 
members.  For  example,  information  related  to  AIS  costs  and  benefits  is 
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obtained  from  the  AIS  program  manager  and  provided  to  the  MAISRC  staff  in 
the  Office  of  the  Director,  Program  Analysis  and  Evaluation,  for  review  and,  if 
appropriate,  validation.  The  MAISRC  staff  performs  most  of  the  "leg  work" 
involved  in  the  MAISRC  oversight  process. 

The  MAISRC  Oversight  Process.  The  MAISRC  oversight  process  begins  with 
identification  of  a  major  AIS.  The  Principal  Staff  Assistant  for  the  functional 
area  supported  by  the  AIS  usually  identifies  an  AIS  for  MAISRC  oversight; 
however,  the  MAISRC  may  become  aware  of  a  major  AIS  through  other 
means,  such  as  through  information  technology  budget  reviews. 

Major  AIS  program  managers  submit  quarterly  status  reports  to  the  MAISRC. 
The  MAISRC  staff  uses  the  quarterly  reports  to  monitor  the  status  of  the  AIS 
programs  and  to  identify  potential  issues. 

Either  the  major  AIS  program  manager  or  the  MAISRC  may  initiate  a  MAISRC 
review.  The  MAISRC  staff  reviews  documentation  submitted  by  the  major  AIS 
program  managers,  identifies  program  deficiencies,  issues,  or  concerns,  and 
tries  to  resolve  problems  with  the  AIS  program  manager.  The  MAISRC  staff 
prepares  a  synopsis  of  the  AIS  program's  history,  status,  and  unresolved  issues 
and  other  information  pertinent  to  the  major  AIS  being  reviewed.  The  synopsis 
is  provided  to  appropriate  MAISRC  members  before  the  MAISRC  formally 
convenes. 

The  results  of  the  MAISRC  review  are  documented  in  a  System  Decision 
Memorandum  (the  Memorandum).  The  Memorandum  reflects  the  milestone 
approval,  if  granted,  and  any  specific  requirements  or  direction  given  to  the 
major  AIS  program  manager  and  specifies  the  exit  criteria  that  must  be 
accomplished  in  the  next  life-cycle  phase. 
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Appendix  C.  Major  Automated  Information 

Systems 

(Subject  to  MAISRC  Review  as  of  August  1994) 


Name  of  Svstem 

Estimated 

Life-Cycle 

Costs1 

room 

FY  1994 
Program 
Costs1 
(0001 

FY  1995 
Program 
Costs1 

toooi 

LCM  Review 
Tvoe/Date  2 

Army 

Army  Food  Management  Information 
System3 

$  1,976,000 

$  11,326 

$  6,635 

Milestone  III 
September  1990 

Army  World  Wide  Military  Command 
and  Control  System  Information 
System5 

1,660,7004 

24,141 

26,709 

In-Process 
February  1993 

Department  of  the  Army  Movement 
Management  System  Redesign 

166,  0004 

10,275 

7,416 

Milestones  II/III 
May  19944 

Installation  Support  Module3*  6 

374,400 

21,998 

11,106 

Milestones  II/III 
July  1993 

Joint  Computer  Aided  Acquisition 
and  Logistics 

1,138,7004 

43,0174 

_7 

In-Process 

March  1994 

Personnel  Electronic  Records 
Management  System3 

421,700 

17,598 

10,001 

Milestones  II/III 
July  1993 

Reserve  Component 

System  Automation 

1,872,400 

219,938 

159,894 

In-Process 

June  1994 

Standard  Army 

Maintenance  System3 

358,700 

12,292 

8,368 

None 

Standard  Army  Retail  Supply 

System3 

773,200 

28,286 

18,375 

Milestone  III 
March  1992 

Standard  Installation/Division 
Personnel  System-Ill 

427,700 

24,779 

21,717 

Milestone  II 
October  1991 

Sustaining  Base  Information 

Services 

1,800,000 

60,960 

102,397 

Milestone  I 
February  1992 

See  footnotes  at  end  of  the  table. 
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Appendix  C.  Major  Automated  Information  Systems 


Name  of  Svstem 

Estimated 

Life-Cycle 

Costs1 

room 

FY  1994 
Program 
Costs1 

room 

FY  1995 
Program 
Costs1 

room 

LCM  Review 
Tvoe/Date  2 

Theater  Automated  Medical 

Management  Information  System3 

358,000 

5,271 

5,671 

Milestone  III 

May  1990 

Transportation  Operation  Personal 
Property  Standard  System8 

143.6004 

20,312 

16,300 

In-Process 

(Paper) 

December  19944 

Unit  Level  Logistics  System 

701,800 

18,752 

6,293 

In-Process 

August  1993 

Total 

$12,172,900 

$  518,945 

$  400,882 

Navy 

Electronic  Military  Personnel 

Records  Systems 

200,000 

4,161 

43,029 

None 

Joint  Engineering  Data  Management 
and  Information  Control  System 

561,100 

9,712 

11,511 

Milestone  III 

May  1991 

Naval  Aviation  Logistics  Command 
Management  Information  System8’ 5 

1,695,200 

i 

55,182 

0 

Milestone  III 
(Paper) 

November  19944 

Navy  Source  Data  System3 

2,295,400 

23,467 

25,674 

Milestone  III 
January  1986 

Navy  Tactical  Command  Support 
System 

„7 

0 

156,158 

None 

Primary  Oceanographic  Prediction 
System3 

359,100 

12,387. 

16,843 

Milestones  I/II 
November  1989 

Shipboard  Non-Tactical  Automated 
Data  Processing  Program  III8’  9 

1,410,000 

29,349 

0 

Milestone  III 
(Paper) 

July  19944 

Stock  Point  Automated  Data 

Processing  Equipment 
Replacement/Data  Processing 
Installation  Consolidation8 

2,712,000 

15,212 

16,488 

None 

Total 

$  9,232,800 

$  149,470 

$  269,703 

See  footnotes  at  the  end  of  the  table. 
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Appendix  C.  Major  Automated  Information  Systems 


Name  of  Svstem 

Estimated 

Life-Cycle 

Costs1 

1000) 

FY  1994 
Program 
Costs1 

room 

FY  1995 
Program 
Costs1 
COOOl 

LCM  Review 
Tvpe/Date  2 

Air  Force 

Air  Force  Command  and  Control 

System8 

360,000 

18,941 

6,385 

Milestones  II/III 
December  1988 

Air  Force  Equipment  Management 
System3 

134,900 

8,815 

5,817 

In-Process 
November  1993 

Air  Mobility  Command, 

Command  and  Control 

Information  Processing  System3 

523,300 

31,953 

24,598 

In-Process 

June  1993 

Base-Level  Computer  System 

-7 

20.8764 

18.9864 

None 

Combat  Ammunition  System3 

450,600 

17,884 

10,197 

In-Process 

July  1993 

Core  Automated  Maintenance  System/ 
Reliability  Maintainability 
Information  System3 

606,000 

23,962 

20,051 

In-Process 
March  1994 

Defense  Civilian  Personnel  Data  System 

130, 0004 

9,0294 

11.1574 

In-Process 
(Ongoing  as  of 
January  1995)4 

Depot  Maintenance  Management 
Information  System 

575,100 

2,264 

2,168 

Milestone  III 
June  1994 

Fuels  Automated  Management  System3 

154,230 

4,344 

2,224 

None 

Global  Transportation  Network 

476,100 

17,160 

28,621 

In-Process 

April  1993 

Integrated  Data  Strategy  Project8 

„7 

14,800 

19,000 

None 

Integrated  Maintenance  Data  System 

„7 

0 

„7 

None 

Personnel  Concept  III3 

475,000 

6,884 

4,701 

Milestone  III 
May  1990 

Requirements  Data  Bank3 

558,230 

10,948 

13,279 

Milestone  III 
July  1993 

See  footnotes  at  the  end  of  the  table. 
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Appendix  C.  Major  Automated  Information  Systems 


Name  of  System 

Estimated 

Life-Cycle 

Costs1 

(000) 

FY  1994 
Program 
Costs1 
(000) 

FY  1995 
Program 
Costs1 
(000) 

Air  Force 

Strategic  War  Planning  System/ 
TRIAD  Computer  System 

683,260 

69,115 

61,464 

Total 

$  5,126,720 

$  256,975 

$  228,648 

Office  of  the  Secretary  of  Defense 

Composite  Health  Care  System 

2,131,200 

179,554 

193,830 

Defense  Medical  Logistics  Standard 
Support  Program 

176,400 

13,789 

18,804 

High  Performance  Computing 
Modernization  Plan 

2,000,0004 

146, 1004 

183.0004 

Theater  Medical  Information  System 

„7 

„7 

__7 

Total 

$  4,307,600 

$  339,443 

$  395,634 

Defense  Commissary  Agency 

Defense  Commissary  Information 
System 

386,300 

2,802 

11,523 

Defense  Commissary  Point  of  Sale 
System 

597,000 

16,608 

15,762 

Total 

$  983,300 

$  19,410 

$  27,285 

Defense  Finance  and  Accounting  Service 

Defense  Joint  Military  Pay  System 

679,000 

45,778 

39,869 

Total 

$  679,000 

$  45,778 

$  39,869 

See  footnotes  at  end  of  the  table. 


LCM  Review 
Tvne/Date  2 


In-Process 
January  1994 


Milestone  III 
May  1992 

Milestone  0 
November  1993 

Milestone  I 
(Paper) 

December  19944 
None 


Milestone  I 
September  1993 

None 


None 
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Appendix  C.  Major  Automated  Information  Systems 


Name  of  System 

Estimated 

Life-Cycle 

Costs1 

('000) 

FY  1994 
Program 
Costs1 
1000) 

FY  1995 
Program 
Costs1 
(000) 

LCM  Review 
Tvoe/Date2 

Defense  Information  Systems 
Agency 

Defense  Information  System  Network 

140,500 

30,419 

27,449 

Milestone  0 
(Ongoing  as  of 
January  1995)4 

Defense  Message  System 

266,500 

7,711 

34,288 

In-Process 
December  19944 

Global  Command  and  Control  System 

-7 

__7 

„ 7 

In-Process 

June  1994 

Total 

$  407,000 

$  38,130 

$  61,737 

Defense  Logistics  Agency 

Automated  Document  Conversion 
System 

..7 

14.0004 

30,0004 

In-Process 

(Paper) 

March  19944 

Distribution  Standard  System 

1,100,000 

44,681 

52,145 

In-Process 

June  1994 

Defense  Automatic  Addressing  System 
Automated  Data  Processing 
Replacement  and  Modernization 
Program3 

185, 0004 

1.8664 

2,8004 

Milestone  III 
October  1993 

Integrated  Contract  Administration 

__7 

5,050 

14,600 

None 

Support 

Total  $  1,285,000  $  65,597  $  99,545 


See  footnotes  at  end  of  the  table. 
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Appendix  C.  Major  Automated  Information  Systems 


Name  of  Svstem 

Estimated 

Life-Cycle 

Costs1 

room 

FY  1994 
Program 
Costs1 
(000) 

FY  1995 
Program 
Costs1 

roooi 

LCM  Review 
Tvtie/Date2 

Joint  Logistics  Systems  Center 

Depot  Maintenance  Standard  System 

__7 

„7 

..7 

In-Process 
March  1993 

Materiel  Management  Standard  System 

__7 

Ij 

„7 

In-Process 
March  1993 

Total 

$  0 

1  0 

$  0 

Grand  Total  for  the  53  systems  $34,194,32010  $1,433,748* 1  $1,523,30312 


^ost  data  obtained  from  Exhibit  43,  FY  1996  DoD  Budget  Estimate,  unless  otherwise 
noted. 

information  obtained  from  August  1994  Office  of  the  Assistant  Secretary  of  Defense 
(Command,  Control,  Communications,  and  Intelligence)  memorandum  designating 
major  AISs  subject  to  MAISRC  and  Component  council  reviews,  unless  otherwise 
noted. 

3Major  AIS  oversight  review  delegated  to  the  Component  council, 
information  obtained  from  MAISRC  action  officers. 

5To  be  merged  into  the  Army  Global  Command  and  Control  System. 

6System  to  be  combined  with  Sustaining  Base  Information  System. 

7Cost  data  not  available. 

8Not  considered  a  major  AIS.  System  will  be  deleted  from  the  Office  of  the  Assistant 
Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence) 
memorandum  designating  major  AISs  subject  to  MAISRC  and  Component  council 
review. 

9System  to  be  combined  with  Navy  Tactical  Command  Support  System. 

10Excludes  costs  for  10  major  AISs  because  cost  data  were  not  available. 
uExcludes  costs  for  four  major  AISs  because  cost  data  were  not  available. 

^Excludes  costs  for  six  major  AISs  because  cost  data  were  not  available. 
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Appendix  D.  Prior  Audits  of  Major  Automated 

Information  Systems 


Government 

Organization 

Report 

Number 

Report  Title 

GAO1 

AIMD-94-61 
(OSD  Case 

No.  9672) 

"Medical  ADP  [Automated  Data  Processing] 
Systems:  Defense's  Tools  and  Methodology  for 
Managing  CHCS  [Composite  Health  Care  System] 
Performance  Need  Strengthening,"  July  15,  1994 

GAO 

IMTEC-93-11 
(OSD  Case 

No.  9269) 

"Composite  Health  Care  System:  Outpatient 
Capability  is  Nearly  Ready  for  Worldwide 
Deployment,"  December  15,  1992 

GAO 

IMTEC-93-03 
(OSD  Case 

No.  9290) 

"Air  Force  ADP:  Lax  Contract  Oversight  Led  to 
Waste  and  Reduced  Competition," 

November  19,  1992 

GAO 

IMTEC -92-69 
(OSD  Case 

No.  9167) 

"ADP  Procurement:  Prompt  Navy  Action  Can 
Reduce  Risks  to  SNAP  III  [Shipboard 

Non-Tactical  ADP  Program  III]  Implementation," 
September  29,  1992 

GAO 

IMTEC-92-54 
(OSD  Case 

No.  9064) 

"Medical  ADP  Systems:  Composite  Health  Care 
System  is  Not  Ready  to  be  Deployed," 

May  20,  1992 

GAO 

IMTEC-92-25 
(OSD  Case 

No.  8838) 

"Defense  ADP:  Lessons  Learned  From 
Development  of  Defense  Distribution  System, " 
March  20,  1992 

GAO 

IMTEC-91-54 
(OSD  Case 
No.  8741) 

"Defense  ADP:  A  Coordinated  Strategy  is  Needed 
to  Implement  the  Computer-Aided  Acquisition 
Logistics  Support  Initiatives,"  September  13,1991 

GAO 

IMTEC-91-47 
(OSD  Case 
No.  8780) 

"Medical  ADP  Systems:  Changes  In  Composite 
Health  Care  System's  Deployment  Strategy  are 
Unwise,"  September  30,  1991 

General  Accounting  Office 


Appendix  D.  Prior  Audits  of  Major  Automated  Information  Systems 


Government 

Organization 

Report 

Number 

Renort  Title 

GAO 

IMTEC-90-66 
(OSD  Case 

No.  8489) 

"Army  Automation:  Decisions  Needed  on 
SIDPERS  III  [Standard  Installation/Division 
Personnel  System-Ill]  Before  Further 
Development,"  September  5,  1990 

GAO 

IMTEC-90-46 
(OSD  Case 

No.  8397) 

"Air  Force  ADP:  Depot  Maintenance  System 
Development  Risks  Are  High," 

May  25,  1990 

GAO 

IMTEC-90-42 
(OSD  Case 

No.  8277-A) 

"Medical  ADP  Systems:  Composite  Healthcare 
System— Defense  Faces  a  Difficult  Task," 

March  15,  1990 

GAO 

IMTEC-90-36 
(OSD  Case 

No.  8328) 

"Automated  Information  System:  Defense's 
Oversight  Process  Should  Be  Improved," 

April  16,  1990 

GAO 

IMTEC-90-25 
(OSD  Case 

No.  8351) 

"  Computer  Systems :  Development  of  N avy '  s 
Source  Data  System  Needs  to  be  Reassessed," 
May  8,  1990 

GAO 

IMTEC-90-22 
(OSD  Case 

No.  8256) 

"Air  Force  ADP:  The  Personnel  Concept  III 
System  is  Not  Ready  For  Deployment," 

February  27,  1990 

GAO 

IMTEC-90-11 
(OSD  Case 
No.  8240) 

"Computer  Acquisition:  Navy's  Aviation 
Logistics  System  Not  Ready  For  Deployment, 
February  9,  1990 

GAO 

IMTEC-90-6 
(OSD  Case 

No.  8232) 

"Air  Force  ADP:  Systems  Funded  Without 
Adequate  Cost/Benefit  Analysis, "  (Includes 
Reliability  and  Maintainability  Information 
System)  December  28,  1989 

IG,  DoD2 

95-141 

"Report  on  the  Controls  Over  U.S.  Army  Funds 

for  the  Army  Global  Command  and  Control 
System, "  March  9,  1995 


inspector  General,  Department  of  Defense 


( 
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Appendix  D.  Prior  Audits  of  Major  Automated  Information  Systems 


Government 

Organization 

Report 

Number 

Report  Title 

IG,  DoD 

95-084 

"Report  on  the  Hotline  Allegations  Concerning  a 
Request  for  Proposal  for  the  Defense  Message 
System,"  January  26,  1995 

Army 

SR  93-206 

"Survey  of  Implementation  of  Major  Automated 
Information  Systems  Review  Councils  at  Major 
Commands,"  February  3,  1993 

Army 

WR  93-751 

"Theater  Army  Medical  Management  Information 
System  Contracts  (TAMMIS),"  December  7,  1992 

Army 

SR  93-301 

"Contract  Administration  Program  Management 
Office  Reserve  Component  Automation  System 
(RCAS),"  November  10,  1992 

Army 

SR  92-15 

"Standard  Army  Retail  Supply  System,"  June  2, 
1992 

Army 

SR  92-301 

"Review  of  Planning  For  Fielding  the  Reserve 
Component  Automation  Systems 
(RCAS),"  January  10,  1992 

Army 

SR  92-300 

"Review  of  Planning  For  Testing  and  Evaluating 
the  Reserve  Component  Automation  System 
(RCAS),"  October  1,  1991 

Navy 

005-N-94 

"Stock  Point  Automated  Processing  Replacement 
Project  (SPAR),"  November  26,  1993 

Navy 

009-C-93 

"Naval  Aviation  Logistics  Command  Management 
Information  System  (NALCOMIS)," 

November  23,  1993 

Navy 

015-C-93 

"Shipboard  Non-Tactical  Automated  Data 
Processing  Program  (SNAP  III),"  January  5, 

1993. 

Navy 

052-N-92 

"Engineering  Data  Management  Information  and 
Control  System  (EDMICS),"  May  1,  1992. 
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Appendix  D.  Prior  Audits  of  Major  Automated  Information  Systems 


Government 

Oreanization 

Report 

Number 

ReDort  Title 

Air  Force 

92066008 

Review  of  the  Design  and  Development  Activities 
for  the  Depot  Maintenance  Management 
Information  System  (DMMIS),"  August  13,  1993 

Air  Force 

92054002 

"Review  of  Combat  Ammunition  System  (CAS)," 
November  30,  1992 

Air  Force 

91066032 

"Review  of  Reliability  and  Maintainability 
Management  Information  System  Contract 
Management  Activities, "  August  13,  1992 

Air  Force 

91066018 

"Review  of  Depot  Maintenance  Management 
Information  Systems  Contract  Management, " 

July  8,  1992 

Air  Force 

91054020 

"Audit  Followup,  Evaluation  of  the  Core 
Automated  Maintenance  System  (CAMS)  Status 
and  Inventory  Subsystem  Application  Controls," 
May  28,  1992 

Air  Force 

91066028 

"Review  of  the  Stock  Control  and  Distribution 
Expected  Benefits,"  September  6,  1991 

Air  Force 

0215412 

"Review  of  Tactical  Air  Forces  Wing  Command 
and  Control  System  Program,"  September  27, 

1991 

Air  Force 

9195424 

Personnel  Concept  III  (PC  III)  Security,"  June  12, 
1991 

Air  Force 

0196625 

"Computer-Aided  Acquisition  and  Logistics 
Support  (CALS)  Implementation  Activities, " 

April  16,  1991 

Air  Force 

0196618 

"Review  of  Internal  Controls  for  Financial 
Management  Information  within  Air  Force 
Logistics  Command's  Logistics  Management 
Systems,"  April  1,  1991. 

Air  Force 

0215411 

"Air  Force  Command  and  Control  Systems 
(AFC2S)  Development,"  December  31,  1990 
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Government 

Organization 

Air  Force 

Air  Force 

Air  Force 

Air  Force 

Air  Force 


Appendix  D.  Prior  Audits  of  Major  Automated  Information  Systems 


Report 

Number  .Report  Title 

019661 1  "Review  of  Design  and  Development  Activities  for 
the  Stock  Control  and  Distribution  System, " 
December  31,  1990 

0196624  "Review  of  Depot  Maintenance  Management 
Information  System  II  (DMMIS  II) 

Implementation  Activities, "  December  7,  1990 

9196620  "Review  of  Contract  Management  Activities  for 
the  Requirements  Data  Bank  (RDB)  Program, " 
November  23,  1990 

9195422  Review  of  Base-Level  Planning  for  Second-Year 
Personnel  Concept  III  (PC  III)  Implementation 
Sites,"  May  14,  1990 

9215411  "Review  of  the  Air  Force  Command  and  Control 
Systems  (AFC2S)  Program,"  March  16,  1990 
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Appendix  E.  Summary  of  Potential  Benefits 

Resulting  From  Audit 

Recommendation  Amount  and/or 

Reference  Description  of  Benefit  Type  of  Benefit 


1. 

Economy  and  Efficiency. 

Strengthens  effectiveness  of 

MAISRC  process  and  integrates 
process  into  the  DoD  Corporate 
Information  Management  Initiative. 

Nonmonetary. 

2. 

Economy  and  Efficiency.  Expands 
MAISRC  role  to  support  the  DoD 
migration  strategies. 

Nonmonetary. 

3. 

Economy  and  Efficiency.  Revises 

DoD  life-cycle  management 
guidance  for  provisions  in 
Recommendations  4.  through  7. 

Nonmonetary. 

4. 

Economy  and  Efficiency  and 
Management  Control.  Establishes 
procedures  to  oversee  major  AIS 
life-cycle  costs  and  benefits. 

Undeterminable. 
Amount  of  funds  put 
to  better  use  can  be 
determined  upon 
validating  costs  and 
benefits. 

5. 

Economy  and  Efficiency  and 
Management  Control.  Provides 
direction  for  the  test  and  evaluation 
certification  and  cost  and  benefit 
validation  of  incrementally 
developed  AISs. 

Nonmonetary. 

6. 

Economy  and  Efficiency.  Provides 
direction  for  new  AIS  development 
methodologies. 

Nonmonetary. 
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Appendix  E.  Summary  of  Potential  Benefits  Resulting  From  Audit 


Recommendation 

Reference 


7. 


8. 


Description  of  Benefit 


Economy  and  Efficiency. 
Establishes  key  information 
requirements  for  each  type  of 
MAISRC  review. 

Economy  and  Efficiency  and 
Management  Control.  Establishes 
criteria  to  provide  MAISRC 
oversight  of  command,  control, 
communications,  and  intelligence 
major  automated  information 
systems. 


Amount  and/or 
Type  of  Benefit 


Nonmonetary. 


Nonmonetary. 
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Appendix  F.  Organizations  Visited  or  Contacted 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  (Comptroller),  Washington,  DC 
Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence),  Washington,  DC 

Director,  Program  Analysis  and  Evaluation,  Washington,  DC 
Director,  Operational  Test  and  Evaluation,  Washington,  DC 
Principal  Deputy  Under  Secretary  of  Defense  for  Acquisition  and  Technology, 
Washington,  DC 

Deputy  Under  Secretary  of  Defense  for  Logistics,  Washington,  DC 


Department  of  the  Army 

Director  of  Information  Systems  for  Command,  Control,  Communications,  and 
Computers,  Washington,  DC 


Department  of  the  Navy 

Naval  Information  Systems  Management  Center,  Arlington,  VA 


Department  of  the  Air  Force 

Deputy  Assistant  Secretary  of  the  Air  Force  (Communications,  Computers,  and 
Support  Systems),  Washington,  DC 


Other  Defense  Organizations 

Director,  Directorate  for  Command,  Control,  Communication  and  Computer  Systems, 
Office  of  the  Joint  Staff,  Washington,  DC 
Director,  Directorate  for  Force  Structure,  Resources  and  Assessment,  Office  of  the 
Joint  Staff,  Washington,  DC 
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Appendix  F.  Organizations  Visited  or  Contacted 


Defense  Agencies 

Defense  Finance  and  Accounting  Service,  Headquarters,  Arlington,  VA 
Defense  Information  Systems  Agency,  Arlington,  VA 

Center  for  Integration  and  Interoperability,  Alexandria,  VA 
Defense  Information  System  Programs  Organization,  Alexandria,  VA 
Defense  Information  Technology  Procurement  Organization,  Arlington,  VA 
Defense  Logistics  Agency,  Alexandria,  VA 


Non-Defense  Federal  Organization 

General  Accounting  Office,  Washington,  DC 
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Appendix  G.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 

Principal  Deputy  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Deputy  Under  Secretary  of  Defense  (Acquisition  Reform) 

Director,  Defense  Logistics  Studies  Information  Exchange 
Under  Secretary  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Director,  Program  Analysis  and  Evaluation 
Under  Secretary  of  Defense  (Personnel  and  Readiness) 

Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence) 
Deputy  Assistant  Secretary  of  Defense  (Acquisition) 

Director,  Operational  Test  and  Evaluation 
Assistant  to  the  Secretary  of  Defense  (Public  Affairs) 

Director,  Joint  Staff 


Department  of  the  Army 

Auditor  General,  Department  of  the  Army 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Navy 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Air  Force 
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Other  Defense  Organizations 

Director,  Defense  Commissary  Agency 
Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Finance  and  Accounting  Service 
Director,  Defense  Information  Systems  Agency 
Director,  Defense  Logistics  Agency 
Director,  National  Security  Agency 

Inspector  General,  National  Security  Agency 


Non-Defense  Federal  Organizations  and  Individuals 

Office  of  Management  and  Budget 

Technical  Information  Center,  National  Security  and  International  Affairs  Division, 
U.S.  General  Accounting  Office 

Chairman  and  ranking  minority  member  of  each  of  the  following  congressional 
committees  and  subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 
House  Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal 
Justice,  Committee  on  Government  Reform  and  Oversight 
House  Committee  on  National  Security 

House  Subcommittee  on  Military  Readiness,  Committee  on  National  Security 
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Part  IV  -  Management  Comments 


VS 


Assistant  Secretary  of  Defense  (Command, 
Control,  Communications,  and  Intelligence) 
Comments 


OFFICE  OF  THE  ASSISTANT  SECRETARY  OF  DEFENSE 
6000  DEFENSE  PENTAGON 
WASHINGTON,  D.C.  20301-6000 

0  8  MAY  1995 


COMMAND.  CONTROL. 
COMMUNICATION 
AND  INTELLIGENCE 


MEMORANDUM  FOR  DIRECTOR,  READINESS  AND  OPERATIONAL  SUPPORT 
DIRECTORATE,  DODIG 

SUBJECT:  Audit  Report  on  the  Oversight  Process  of  the  Major  Automated  Information  System 
Review  Council  (Project  No.  4RE-5025.01) 


We  appreciate  the  opportunity  to  comment  on  the  subject  report.  Many  of  the 
recommendations  in  the  report  could  improve  MAISRC  oversight.  While  the  current  AIS  life- 
cycle  management  directives  are  intended  to  achieve  the  goals  of  most  of  your  recommendations, 
we  agree  that  improvements  and  clarifications  would  be  helpful.  As  indicated  in  the  attached 
response,  a  number  of  initiatives  are  underway  that  will  result  in  revisions  to  the  directives. 

These  revisions  will  achieve  many  of  the  report’s  recommendations. 

As  requested  by  your  memorandum  of  March  8,  1995,  we  have  addressed  each 
recommendation  by  indicating  either  concurrence  or  nonconcurrence,  provided  the  specific 
reasons  for  each  nonconcurrence,  and  described  planned  corrective  actions,  where  appropriate. 

Questions  regarding  the  attached  response  may  be  directed  to  my  action  officer,  Dave 
Mullins,  at  703-756-4987. 


^nthoijy'M.  Valletta 
Deputy  Assistant  Secretary  of  Defense 
(C3I  Acquisition) 


Attachment 


ASD(C3I) 

DASD(IM) 
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Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 

Intelligence)  Comments 


Final  Report 
Reference 


DASD(C3I  ACQUISITION)  RESPONSE  TO 
AUDIT  REPORT  ON  THE  OVERSIGHT  PROCESS  OF  THE  MAISRC 
(PROJECT  NO.  4RE-5025.01) 

Recommendation  1:  Revise  DoD  Directive  8120.1,  “Life-Cycle  Management  of  Automated 
Information  Systems,”  January  14, 1993;  DoD  Instruction  8120.2,  “Automated 
Information  Systems  Life-Cycle  Management  Process,  Review,  and  Milestone  Approval 
Procedures,”  January  14, 1993;  and  DoD  Manual  7920.2M,  “Automated  Information 
Systems,  Life-Cycle  Management  Manual,”  March  1990,  to  reflect  the  provisions  in 
Recommendations  2  through  7. 

OSD(C3I)  Response: 

Concur.  As  indicated  below,  the  revisions  to  DoD  Manual  7920.2-M,  which  will  be 
published  as  DoD  Manual  8102.2-M,  will  address  some  of  the  concerns  expressed  in  this 
report.  All  of  the  above-mentioned  directives  and  instructions  will  be  revised  as  the  result 
of  other  initiatives,  such  as  the  MAISRC  Functional  Process  Improvement  Project  and  the 
December  9,  1994  report  on  “Reengineering  the  Acquisition  Oversight  and  Review 
Process.” 

Required  Actions:  None,  other  than  as  indicated  below. 

Recommendation  2:  Specify  procedures  that  will  involve  the  Major  Automated 
Information  Systems  Review  Council  staff  in  the  ongoing  efforts  of  the  DoD  Corporate 
Information  Management  Initiative. 

OSD(C3I)  Response: 

Nonconcur.  The  management  structure  and  focus  for  DoD  Corporate  Information 
Management  initiatives  is  in  four  areas:  common  information  systems;  shared,  standard 
data;  reengineered  business  processes;  and  a  computer  and  communications 
infrastructure.  The  OSD  functional  Principal  Staff  Assistants  (PSAs),  by  Deputy 
Secretary  direction,  are  to  accomplish  the  first  three  areas.  The  Assistant  Secretary  of 
Defense  for  Command,  Control,  Communications  and  Intelligence  (ASD(C3I))  is  to 
implement  a  flexible,  world- wide  computer  and  communications  infrastructure.  The 
DoD  has  established  the  Enterprise  Integration  Corporate  Management  Council  to 
provide  senior  leadership  for  management  and  oversight  of  implementation  of  cross¬ 
functional  integration  issues  and  technical  infrastructure  solutions,  including  specific 
information  systems  within  the  Defense  Information  Infrastructure.  The  MAISRC  staff 
participates  in  this  process  whenever  the  CIM  initiatives  become  major  automated 
information  systems. 

Required  Actions:  None 


Renumbered 
as  Recommei 
dation  3. 


Renumbered 
as  Recommer 
dation  1. 
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Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence)  Comments 


Final  Report 
Reference 


Renum¬ 
bered  as 
Recom¬ 
menda¬ 
tion  2. 


Revised. 


Revised 

and 

Renum¬ 
bered  as 
Recom¬ 
menda¬ 
tion  6. 


Recommendation  3:  Specify  procedures  for  the  Major  Automated  Information  System 
Review  Council  staff  to  perform  assessments  of,  and  concurrent  with,  the  identification  and 
development  of  each  migratory  major  automated  information  system  that  will  be  used  on  a 
DoD-wide  basis. 

OSD(C3I)  Response: 

Nonconcur.  Current  life-cycle  management  policies  already  apply  to  migration  systems, 
and  when  those  systems  meet  the  major  AIS  threshold,  their  development  and 
management  are  subject  to  OSD  MAISRC  oversight.  By  Deputy  Secretary  direction,  the 
identification  of  migration  systems  is  to  be  made  by  the  functional  OSD  PSA  based  on 
evaluation  criteria  provided  by  the  ASD(C3I)  and  supported  by  DISA’s  integration 
checklist  for  migration  assessment.  The  ASD(C3I)  reviews  and  approves  each  migration 
system  proposed  by  the  PSAs.  Currently,  almost  half  of  the  43  major  AIS  programs 
under  OSD  MAISRC  oversight,  are  migration  systems. 

Required  Actions:  None 


Recommendation  4:  Designate  the  Director,  PA&E,  to  validate  the  costs  and  benefits  of  a 
major  AIS  throughout  its  life  cycle. 

OSD(C3I)  Response: 

Concur.  DoD  Directive  8120.1  (paragraph  E.4.)  and  DoD  Instruction  8120.2  (paragraph 
E.4.)  already  give  the  Director,  PA&E,  the  responsibility  to  validate  the  costs  and  benefits 
of  a  major  AIS  throughout  its  life  cycle.  However,  as  indicated  in  our  response  to 
Recommendation  6  below,  clarifying  guidance  will  be  issued  by  PA&E  and  in  DoD 
Manual  8120.2-M 

Required  Actions:  See  Recommendation  6  below. 


Recommendation  5:  Establish  procedures  and  criteria  to  oversee  the  cost,  schedule,  and 
performance  of  major  AISs  developed  in  increments. 

OSD(C3I)  Response: 

Concur  that  DoD  life-cycle  management  (LCM)  directives  should  include  procedures  and 
criteria  for  overseeing  incrementally  developed  AISs.  Currently,  DoD  Directive  8120.1 
(paragraph  F.3.)  requires  programs  employing  a  tailored  strategy  to  obtain  “up-front” 
approval  for  that  strategy  from  the  milestone  decision  authority  before  program 
management  proceeds  to  the  next  phase.  Also,  the  program  manager  must  obtain  written 
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Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 

Intelligence)  Comments 


Final  Report 
Reference 


approval  for  the  strategy  in  the  system  decision  memorandum  (SDM).  These 
requirements  result  in  SDM  instructions  to  the  program  managers  on  what  cost-and- 
benefit-related  analyses  must  be  accomplished  within  specific  time  frames. 

Required  Actions:  We  will  consider  clarifying  the  procedures  for  the  oversight  of 
incrementally  developed  AISs  as  part  of  the  planned  rewrites  of  DoDD  8120.1  and  DoDI 
8120.2.  These  rewrites,  which  will  result  from  the  results  of  the  MAISRC  Re- 
Engineering  Project  and  other  reform  initiatives,  are  scheduled  to  begin  in  July  1995. 

Recommendation  6:  Specify  life-cycle  management  requirements  for  validating  costs  and 
benefits  and  for  performing  operational  tests  and  evaluations  of  major  automated 
information  systems  developed  in  increments.  Determine  the  life-cycle  management 
requirements  in  coordination  with  the  Director,  Program  Analysis  and  Evaluation,  and  the 
Director,  Operational  Test  and  Evaluation. 

OASD(C3I)  Response: 

Concur.  The  finding  that  supports  this  recommendation  states  that  operational  testing  is 
performed  only  on  completed  AISs,  hot  on  AISs  developed  and  deployed  in  increments. 
This  is  not  the  case.  A  typical  example  of  operational  testing  for  a  major  AIS  being 
implemented  incrementally  is  the  Composite  Health  Care  System  (CHCS),  which 
achieved  Milestone  III  approval  in  October  1992,  after  extensive  operational  test  and 
evaluation  for  its  initial  outpatient  release.  Since  then,  several  CHCS  releases 
(increments)  have  been  developed  and  deployed  after  thorough  and  comprehensive 
operational  testing.  Further,  all  follow-on  system  increments  are  required  to  go  through 
adequate  operational  testing  prior  to  fielding. 

Paragraph  E.7.  of  both  DoD  Directive  8120.1  and  DoD  Instruction  8120.2  assign 
responsibly  to  the  Director,  OT&E  to  assess  and  validate  major  AIS  program  compliance 
with  applicable  operational  test  and  evaluation  planning  policies  and  procedures.  The 
Director,  OT&E  also  approves  the  test  and  evaluation  master  plan  (TEMP)  and 
operational  test  plans;  monitors  operational  test  and  evaluation,  in  accordance  with  DoD 
Instruction  5000.2;  and  provides  the  test  and  evaluation  results  to  the  MDA.  Specific 
procedures  for  conducting  operational  testing  for  AIS  programs  developed  and  deployed 
in  increments  are  being  written  and  will  be  annexed  to  the  final  DOD  Manual  8120.2-M, 
“Automated  Information  System  Life  Cycle  Management  Manual.” 

As  indicated  above,  Paragraph  E.4.  of  both  DoD  Directive  8120.1  and  DoD  Instruction 
8120.2  outline  the  responsibilities  of  the  Director,  PA&E.  These  responsibilities  include 
the  review  and  validation,  at  appropriate  LCM  reviews,  of  major  AIS  program  cost 
estimates,  life-cycle  cost  estimates,  independent  cost  estimates,  benefit  analyses,  and 
functional  economic  analyses  (FEAs). 
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Renumbered 
as  Recommer 
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Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence)  Comments 


An  incremental  program  is  generally  characterized  by  acquisition,  development,  and 
deployment  of  functionality  through  a  number  of  clearly  defined  system  “increments” 
that  essentially  stand  on  their  own.  (Some  of  the  investment  costs  for  the  initial 
increments,  particularly  for  hardware,  may  support  later  increments.)  DoDI  8 1 20.2 
recognizes  that  systems  developed  in  increments  cannot  be  overseen  effectively  using  the 
traditional  milestone  approval  process.  Hence,  it  states  that  “depending  on  the  selected 
program  strategy,  combined  or  repeated  milestone  decision  points  and  associated 
activities  within  the  life-cycle  phase  may  be  required.  The  number  of  replicated  decision 
points,  and  how  increments  between  those  decision  points  are  reviewed,  shall  be  specified 
in  the  proposed  program  strategy  presented  at  Milestone  I.” 

PA&E  recognizes  that  the  costs  and  benefits  for  an  incremental  program  are  difficult  to 
estimate  for  the  program  as  a  whole,  because  different  increments  are  at  different  stages 
of  definition,  development,  and  deployment.  The  cost  and  benefit  estimates  are  normally 
most  accurate  and  verifiable  for  those  increments  that  are  closest  to  deployment.  Future 
increments,  which  account  for  a  large  portion  of  the  claimed  benefits,  fluctuate  regularly 
in  terms  of  the  functionality  that  they  will  encompass.  Thus,  PA&E  recommends  that  the 
cost  and  benefit  analyses  for  an  incremental  program  undergoing  a  MAISRC  milestone 
review  be  focused  on  the  increment  or  increments  that  are  in  phase  with  that  milestone. 
For  the  Sustaining  Base  Information  Services  (SBIS)  program,  for  example,  which  is 
requesting  a  Milestone  II/III  review  later  this  year,  the  MAISRC  will  review  the  first  two 
increments  thoroughly  in  term  of  costs  and  benefits.  On  the  basis  of  that  review,  the 
MAISRC  can  choose  to:  i)  hold  repeated  Milestone  II/III  reviews  for  subsequent 
increments,  ii)  delegate  the  milestone  approval  process  for  subsequent  increments  to  the 
Service  and  hold  periodic  in-process  reviews,  or  iii)  delegate  all  future  reviews  to  the 
Service.  The  drawback  to  this  approach  is  that,  because  of  an  initial  investment  in 
hardware  and  system  configuration,  the  return  on  investment  for  early  increments  may 
not  be  as  favorable  as  for  the  program  as  a  whole.  This  factor  must  be  taken  into  account 
in  the  cost/benefit  review.  PA&E  has  provided  draft  procedures  for  developing  an  AIS 
Economic  Analysis  for  publication  in  the  final  DOD  Manual  8120.2-M.  “Automated 
Information  System  Life  Cycle  Management  Manual.”  PA&E  is  also  in  the  process  of 
revising  “DoD  Automated  Information  Systems  (AIS)  Economic  Analysis  (EA)  Guide 
(DAG)  and  plan  to  publish  it  with  C3I  concurrence. 

Required  Actions:  ASD(C3I)  will  publish  DOD  Manual 

8120.2-M,  “Automated  Information  System  Life  Cycle  Management  Manual”  by 

September  1995;  and  Director  PA&E  plans  to  publish  the  DAG  by  October  1995. 


Recommendation  7:  Designate  key  program  management  information,  such  as  validate 
cost-benefits  analyses  and  operational  test  plans,  for  milestone  and  in-process  reviews 
performed  by  the  MAISRC. 

OASD(C3I)  Response 
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Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 

Intelligence)  Comments 


Concur.  DoD  Directive  8120.1  (Paragraph  F.3.)  provides  for  approval  of  tailoring  and 
special  guidance  pertaining  to  each  review  (whether  it  be  a  milestone  review  or  an  in- 
process  review  (IPR)).  DoD  Instruction  8120.2  describes  only  the  nature  of  the 
documentation  for  IPRs  because,  by  design,  IPRs  can  be  conducted  for  various  purposes. 
Thus,  the  specific  purpose  and  documentation  requirements  for  such  reviews  are  stated  in 
the  memorandum  requesting  the  review.  The  final  DOD  Manual  8 1 20.2-M,  “Automated 
Information  System  Life  Cycle  Management  Manual,”  which  will  replace  DoD  Manual 
7920.2,  will  provide  more  detailed  procedures  for  the  conduct  of  AIS  LCM  activities  and 
improved  guidelines  for  the  preparation  and  submission  of  documentation  for  LCM 
reviews. 

Required  Actions.  Publish  DOD  Manual  81 20.2-M,  “Automated  Information  System 
Life  Cycle  Management  Manual”  by  October  1 995. 


Recommendation  8:  Clarify  the  oversight  responsibilities  of  the  Major  Automated 
Information  Systems  Review  Council  for  command  and  control,  communications,  and 
intelligence  major  automated  information  systems  that  have  not  been  exempted  from  DoD 
Directive  8000.1,  “Defense  Information  Management  Program,”  October  27,  1992. 

OSD(C3I)  Response: 

Concur  that  MAISRC  oversight  responsibilities  for  command  and  control  systems  should 
be  clarified.  Currently,  Paragraph  B  of  DoD  Directive  8120.1  defines  the  scope  and 
applicability  of  AISs  subject  to  MAISRC  review.  Paragraph  B.2.  of  DoDD  8120.1  states 
that  the  Directive  applies  to 

“. . .  all  AISs  .  . .  that  support  all  DoD  mission  areas,  except  as  specified  in  subsections 
B.3.  through  B.7.,  below  ”  None  of  the  exceptions  in  subsections  B.3.  through  B.7. 
exempt  command  and  control  systems.  Moreover,  Paragraph  B.2.  goes  on  to  state  that 
the  applicability  of  DoDD  8120.1  “is  not  determined  by  the  applicability  of  the  Brooks 
Act  or  the  Warner  Amendment.”  That  statement  is  intended  to  clarify  that  a  command 
and  control  exemption  under  the  Warner  Amendment  does  not  equate  to  an  exemption 
from  DoDD  8120.1. 

Required  Actions:  The  revisions  to  DoDI  8120.1  and  DoDI  8120.2  scheduled  to  begin 
in  July  1995  will  further  clarify  that  major  AISs  used  for  command  and  control  are 
subject  to  MAISRC  oversight. 
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Audit  Team  Members 


This  report  was  prepared  by  the  Readiness  and  Operational  Support 
Directorate,  Office  of  the  Assistant  Inspector  General  for  Auditing, 
Department  of  Defense. 


Thomas  F.  Gimble 
Salvatore  D.  Guli 
Mary  Lu  Ugone 
James  Hutchinson 
Wesley  Lewis 
Joseph  Hurley 
Nancy  C.  Cipolla 
Cristina  Maria  H.  Giusti 
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